WEBVTT Kind: captions Language: en 00:00:00.530 --> 00:00:04.130 align:start position:0% all<00:00:01.530> right<00:00:01.680> well<00:00:01.890> hello<00:00:02.159> everyone<00:00:02.460> I'm<00:00:03.240> Adam 00:00:04.130 --> 00:00:04.140 align:start position:0% all right well hello everyone I'm Adam 00:00:04.140 --> 00:00:07.120 align:start position:0% all right well hello everyone I'm Adam link<00:00:04.380> I'm<00:00:04.680> an<00:00:04.890> engineer<00:00:05.279> with<00:00:05.520> air<00:00:05.910> swap<00:00:06.299> and 00:00:07.120 --> 00:00:07.130 align:start position:0% link I'm an engineer with air swap and 00:00:07.130 --> 00:00:10.190 align:start position:0% link I'm an engineer with air swap and today<00:00:08.130> we<00:00:08.610> are<00:00:08.880> going<00:00:09.599> to<00:00:09.690> be<00:00:09.750> talking<00:00:10.110> about 00:00:10.190 --> 00:00:10.200 align:start position:0% today we are going to be talking about 00:00:10.200 --> 00:00:14.150 align:start position:0% today we are going to be talking about running<00:00:10.860> a<00:00:11.370> guest<00:00:12.120> node<00:00:12.300> cluster<00:00:12.540> in<00:00:13.160> in 00:00:14.150 --> 00:00:14.160 align:start position:0% running a guest node cluster in in 00:00:14.160 --> 00:00:18.170 align:start position:0% running a guest node cluster in in production<00:00:14.759> Netscape<00:00:16.490> so<00:00:17.490> as<00:00:17.640> was<00:00:17.789> alluded<00:00:18.000> to 00:00:18.170 --> 00:00:18.180 align:start position:0% production Netscape so as was alluded to 00:00:18.180 --> 00:00:20.330 align:start position:0% production Netscape so as was alluded to about<00:00:18.390> 15<00:00:18.779> minutes<00:00:19.109> ago<00:00:19.260> there<00:00:19.590> is<00:00:19.619> no<00:00:19.980> open 00:00:20.330 --> 00:00:20.340 align:start position:0% about 15 minutes ago there is no open 00:00:20.340 --> 00:00:22.519 align:start position:0% about 15 minutes ago there is no open source<00:00:20.609> way<00:00:20.789> to<00:00:20.850> run<00:00:21.090> a<00:00:21.119> guest<00:00:21.420> node<00:00:21.689> at<00:00:21.960> scale 00:00:22.519 --> 00:00:22.529 align:start position:0% source way to run a guest node at scale 00:00:22.529 --> 00:00:24.620 align:start position:0% source way to run a guest node at scale we<00:00:22.830> will<00:00:23.340> be<00:00:23.490> open<00:00:23.760> sourcing<00:00:24.150> our<00:00:24.240> production 00:00:24.620 --> 00:00:24.630 align:start position:0% we will be open sourcing our production 00:00:24.630 --> 00:00:30.009 align:start position:0% we will be open sourcing our production infrastructure<00:00:25.260> in<00:00:25.500> about<00:00:25.740> 20<00:00:26.010> minutes<00:00:26.420> yep 00:00:30.009 --> 00:00:30.019 align:start position:0% 00:00:30.019 --> 00:00:33.350 align:start position:0% so<00:00:31.019> this<00:00:31.470> is<00:00:31.650> the<00:00:31.969> work<00:00:32.969> about<00:00:33.149> three 00:00:33.350 --> 00:00:33.360 align:start position:0% so this is the work about three 00:00:33.360 --> 00:00:35.500 align:start position:0% so this is the work about three developers<00:00:33.780> over<00:00:34.260> about<00:00:34.410> two<00:00:34.710> months<00:00:35.040> to 00:00:35.500 --> 00:00:35.510 align:start position:0% developers over about two months to 00:00:35.510 --> 00:00:38.000 align:start position:0% developers over about two months to develop<00:00:36.510> the<00:00:36.600> infrastructure<00:00:37.020> with<00:00:37.530> a<00:00:37.559> lot<00:00:37.920> of 00:00:38.000 --> 00:00:38.010 align:start position:0% develop the infrastructure with a lot of 00:00:38.010 --> 00:00:39.350 align:start position:0% develop the infrastructure with a lot of the<00:00:38.070> pitfalls<00:00:38.430> that<00:00:38.520> we<00:00:38.670> experienced<00:00:39.180> along 00:00:39.350 --> 00:00:39.360 align:start position:0% the pitfalls that we experienced along 00:00:39.360 --> 00:00:41.360 align:start position:0% the pitfalls that we experienced along the<00:00:39.540> way<00:00:39.660> so<00:00:40.500> it's<00:00:40.649> at<00:00:40.739> least<00:00:40.770> something<00:00:41.100> for 00:00:41.360 --> 00:00:41.370 align:start position:0% the way so it's at least something for 00:00:41.370 --> 00:00:43.340 align:start position:0% the way so it's at least something for you<00:00:41.460> guys<00:00:41.550> to<00:00:41.670> reference<00:00:42.480> as<00:00:42.780> you're<00:00:42.989> building 00:00:43.340 --> 00:00:43.350 align:start position:0% you guys to reference as you're building 00:00:43.350 --> 00:00:46.190 align:start position:0% you guys to reference as you're building a<00:00:43.530> Gotha<00:00:43.800> node<00:00:43.980> cluster<00:00:44.190> or<00:00:44.610> perhaps<00:00:45.570> use<00:00:45.870> on 00:00:46.190 --> 00:00:46.200 align:start position:0% a Gotha node cluster or perhaps use on 00:00:46.200 --> 00:00:51.049 align:start position:0% a Gotha node cluster or perhaps use on your<00:00:46.350> own<00:00:48.170> so<00:00:49.170> at<00:00:49.500> the<00:00:49.649> core<00:00:49.829> this<00:00:50.039> adapts<00:00:50.730> need 00:00:51.049 --> 00:00:51.059 align:start position:0% your own so at the core this adapts need 00:00:51.059 --> 00:00:54.110 align:start position:0% your own so at the core this adapts need to<00:00:51.210> keep<00:00:51.420> state<00:00:51.750> right<00:00:52.590> when<00:00:53.039> you<00:00:53.219> run<00:00:53.430> a<00:00:53.730> DAP 00:00:54.110 --> 00:00:54.120 align:start position:0% to keep state right when you run a DAP 00:00:54.120 --> 00:00:55.369 align:start position:0% to keep state right when you run a DAP your<00:00:54.360> data<00:00:54.539> store<00:00:54.809> is<00:00:54.989> inherently 00:00:55.369 --> 00:00:55.379 align:start position:0% your data store is inherently 00:00:55.379 --> 00:00:57.950 align:start position:0% your data store is inherently distributed<00:00:56.090> and<00:00:57.090> distributed<00:00:57.600> datastore 00:00:57.950 --> 00:00:57.960 align:start position:0% distributed and distributed datastore 00:00:57.960 --> 00:00:59.540 align:start position:0% distributed and distributed datastore means<00:00:58.350> that<00:00:58.500> you<00:00:58.559> need<00:00:58.649> to<00:00:58.710> stay<00:00:58.890> in<00:00:59.070> sync<00:00:59.309> with 00:00:59.540 --> 00:00:59.550 align:start position:0% means that you need to stay in sync with 00:00:59.550 --> 00:01:01.520 align:start position:0% means that you need to stay in sync with your<00:00:59.670> peers<00:01:00.059> to<00:01:00.359> decide<00:01:00.989> what<00:01:01.170> the<00:01:01.260> correct 00:01:01.520 --> 00:01:01.530 align:start position:0% your peers to decide what the correct 00:01:01.530 --> 00:01:04.759 align:start position:0% your peers to decide what the correct view<00:01:01.680> of<00:01:01.800> that<00:01:01.920> data<00:01:02.300> actually<00:01:03.300> is<00:01:03.680> there's<00:01:04.680> a 00:01:04.759 --> 00:01:04.769 align:start position:0% view of that data actually is there's a 00:01:04.769 --> 00:01:06.710 align:start position:0% view of that data actually is there's a whole<00:01:04.890> host<00:01:05.129> of<00:01:05.309> issues<00:01:05.430> that<00:01:05.909> can<00:01:06.090> occur<00:01:06.210> if 00:01:06.710 --> 00:01:06.720 align:start position:0% whole host of issues that can occur if 00:01:06.720 --> 00:01:08.510 align:start position:0% whole host of issues that can occur if your<00:01:07.229> front<00:01:07.590> end<00:01:07.799> or<00:01:08.010> your<00:01:08.159> back<00:01:08.340> and 00:01:08.510 --> 00:01:08.520 align:start position:0% your front end or your back and 00:01:08.520 --> 00:01:11.060 align:start position:0% your front end or your back and application<00:01:09.090> does<00:01:09.840> not<00:01:09.869> properly<00:01:10.530> reflect 00:01:11.060 --> 00:01:11.070 align:start position:0% application does not properly reflect 00:01:11.070 --> 00:01:13.609 align:start position:0% application does not properly reflect what<00:01:11.850> the<00:01:12.000> consensus<00:01:12.600> of<00:01:12.840> the<00:01:12.990> database<00:01:13.320> state 00:01:13.609 --> 00:01:13.619 align:start position:0% what the consensus of the database state 00:01:13.619 --> 00:01:16.160 align:start position:0% what the consensus of the database state actually<00:01:13.860> is<00:01:14.430> and<00:01:14.820> there's<00:01:15.810> nothing<00:01:16.049> worse 00:01:16.160 --> 00:01:16.170 align:start position:0% actually is and there's nothing worse 00:01:16.170 --> 00:01:18.020 align:start position:0% actually is and there's nothing worse than<00:01:16.200> users<00:01:16.710> not<00:01:16.890> trusting<00:01:17.310> your<00:01:17.400> application 00:01:18.020 --> 00:01:18.030 align:start position:0% than users not trusting your application 00:01:18.030 --> 00:01:19.940 align:start position:0% than users not trusting your application because<00:01:18.689> they<00:01:19.049> can't<00:01:19.259> trust<00:01:19.500> the<00:01:19.619> underlying 00:01:19.940 --> 00:01:19.950 align:start position:0% because they can't trust the underlying 00:01:19.950 --> 00:01:22.760 align:start position:0% because they can't trust the underlying data<00:01:20.420> but<00:01:21.420> you<00:01:21.540> also<00:01:21.720> can't<00:01:22.020> sacrifice<00:01:22.259> the 00:01:22.760 --> 00:01:22.770 align:start position:0% data but you also can't sacrifice the 00:01:22.770 --> 00:01:25.130 align:start position:0% data but you also can't sacrifice the responsiveness<00:01:23.430> of<00:01:23.520> your<00:01:23.880> application<00:01:24.450> when 00:01:25.130 --> 00:01:25.140 align:start position:0% responsiveness of your application when 00:01:25.140 --> 00:01:26.359 align:start position:0% responsiveness of your application when you're<00:01:25.290> waiting<00:01:25.619> on<00:01:25.710> an<00:01:25.890> eventually 00:01:26.359 --> 00:01:26.369 align:start position:0% you're waiting on an eventually 00:01:26.369 --> 00:01:29.270 align:start position:0% you're waiting on an eventually consistent<00:01:26.939> database<00:01:28.040> users<00:01:29.040> on<00:01:29.189> the 00:01:29.270 --> 00:01:29.280 align:start position:0% consistent database users on the 00:01:29.280 --> 00:01:30.980 align:start position:0% consistent database users on the Internet<00:01:29.610> are<00:01:29.759> generally<00:01:30.150> not<00:01:30.270> accustomed<00:01:30.900> to 00:01:30.980 --> 00:01:30.990 align:start position:0% Internet are generally not accustomed to 00:01:30.990 --> 00:01:32.960 align:start position:0% Internet are generally not accustomed to waiting<00:01:31.140> five<00:01:31.500> minutes<00:01:31.920> to<00:01:32.159> do<00:01:32.310> a<00:01:32.340> state 00:01:32.960 --> 00:01:32.970 align:start position:0% waiting five minutes to do a state 00:01:32.970 --> 00:01:35.539 align:start position:0% waiting five minutes to do a state change<00:01:33.180> operation<00:01:33.840> and<00:01:34.290> while<00:01:35.130> we<00:01:35.280> in<00:01:35.430> the 00:01:35.539 --> 00:01:35.549 align:start position:0% change operation and while we in the 00:01:35.549 --> 00:01:37.399 align:start position:0% change operation and while we in the industry<00:01:35.670> may<00:01:36.060> be<00:01:36.119> okay<00:01:36.540> with<00:01:36.750> waiting<00:01:36.960> for 00:01:37.399 --> 00:01:37.409 align:start position:0% industry may be okay with waiting for 00:01:37.409 --> 00:01:39.319 align:start position:0% industry may be okay with waiting for transactions<00:01:38.040> to<00:01:38.220> settle<00:01:38.579> if<00:01:38.759> we<00:01:39.119> want<00:01:39.299> to 00:01:39.319 --> 00:01:39.329 align:start position:0% transactions to settle if we want to 00:01:39.329 --> 00:01:43.609 align:start position:0% transactions to settle if we want to actually<00:01:39.810> get<00:01:39.960> mass<00:01:40.170> adoption<00:01:40.530> we<00:01:41.490> need<00:01:41.759> to<00:01:42.619> we 00:01:43.609 --> 00:01:43.619 align:start position:0% actually get mass adoption we need to we 00:01:43.619 --> 00:01:45.560 align:start position:0% actually get mass adoption we need to we do<00:01:43.770> understand<00:01:44.579> the<00:01:44.670> general<00:01:44.939> public<00:01:45.210> wants 00:01:45.560 --> 00:01:45.570 align:start position:0% do understand the general public wants 00:01:45.570 --> 00:01:47.780 align:start position:0% do understand the general public wants which<00:01:45.930> is<00:01:46.200> a<00:01:46.229> lack<00:01:46.680> of<00:01:46.710> leg<00:01:47.159> when<00:01:47.640> they're 00:01:47.780 --> 00:01:47.790 align:start position:0% which is a lack of leg when they're 00:01:47.790 --> 00:01:52.190 align:start position:0% which is a lack of leg when they're doing<00:01:48.149> a<00:01:48.390> state<00:01:48.659> change<00:01:48.840> operation<00:01:51.200> there's 00:01:52.190 --> 00:01:52.200 align:start position:0% doing a state change operation there's 00:01:52.200 --> 00:01:54.499 align:start position:0% doing a state change operation there's two<00:01:52.350> types<00:01:52.590> of<00:01:52.770> data<00:01:53.369> that<00:01:53.640> you<00:01:53.790> need<00:01:54.000> for 00:01:54.499 --> 00:01:54.509 align:start position:0% two types of data that you need for 00:01:54.509 --> 00:01:57.819 align:start position:0% two types of data that you need for state<00:01:54.840> management<00:01:55.399> the<00:01:56.399> first<00:01:57.000> type<00:01:57.329> is<00:01:57.630> the 00:01:57.819 --> 00:01:57.829 align:start position:0% state management the first type is the 00:01:57.829 --> 00:02:00.800 align:start position:0% state management the first type is the balances<00:01:58.829> on<00:01:59.189> your<00:01:59.610> accounts<00:02:00.090> right<00:02:00.360> this<00:02:00.600> is 00:02:00.800 --> 00:02:00.810 align:start position:0% balances on your accounts right this is 00:02:00.810 --> 00:02:02.990 align:start position:0% balances on your accounts right this is the<00:02:01.020> eath<00:02:01.290> that<00:02:01.710> you<00:02:01.890> hold<00:02:02.130> in<00:02:02.430> your<00:02:02.579> wallet 00:02:02.990 --> 00:02:03.000 align:start position:0% the eath that you hold in your wallet 00:02:03.000 --> 00:02:06.380 align:start position:0% the eath that you hold in your wallet and<00:02:03.409> for<00:02:04.409> many<00:02:04.560> depths<00:02:05.070> this<00:02:05.610> is<00:02:05.820> a<00:02:05.850> critical 00:02:06.380 --> 00:02:06.390 align:start position:0% and for many depths this is a critical 00:02:06.390 --> 00:02:08.180 align:start position:0% and for many depths this is a critical piece<00:02:06.540> of<00:02:06.780> data<00:02:06.960> in<00:02:07.290> the<00:02:07.439> user<00:02:07.649> onboarding 00:02:08.180 --> 00:02:08.190 align:start position:0% piece of data in the user onboarding 00:02:08.190 --> 00:02:10.520 align:start position:0% piece of data in the user onboarding process<00:02:08.640> because<00:02:09.479> they<00:02:09.690> may<00:02:09.840> need<00:02:10.050> to<00:02:10.229> swap 00:02:10.520 --> 00:02:10.530 align:start position:0% process because they may need to swap 00:02:10.530 --> 00:02:12.680 align:start position:0% process because they may need to swap their<00:02:10.770> eath<00:02:10.950> for<00:02:11.520> your<00:02:11.760> proprietary<00:02:12.120> DAP 00:02:12.680 --> 00:02:12.690 align:start position:0% their eath for your proprietary DAP 00:02:12.690 --> 00:02:13.380 align:start position:0% their eath for your proprietary DAP token 00:02:13.380 --> 00:02:13.390 align:start position:0% token 00:02:13.390 --> 00:02:16.110 align:start position:0% token in<00:02:13.720> order<00:02:14.020> to<00:02:14.230> use<00:02:14.620> the<00:02:15.400> product<00:02:15.940> that<00:02:16.000> you're 00:02:16.110 --> 00:02:16.120 align:start position:0% in order to use the product that you're 00:02:16.120 --> 00:02:20.100 align:start position:0% in order to use the product that you're creating<00:02:17.040> the<00:02:18.040> second<00:02:18.970> type<00:02:19.480> of<00:02:19.510> information 00:02:20.100 --> 00:02:20.110 align:start position:0% creating the second type of information 00:02:20.110 --> 00:02:22.290 align:start position:0% creating the second type of information is<00:02:20.350> the<00:02:20.380> the<00:02:20.890> state<00:02:21.130> tree<00:02:21.400> this<00:02:22.060> is 00:02:22.290 --> 00:02:22.300 align:start position:0% is the the state tree this is 00:02:22.300 --> 00:02:23.820 align:start position:0% is the the state tree this is information<00:02:22.570> about<00:02:22.930> events<00:02:23.200> within<00:02:23.500> your 00:02:23.820 --> 00:02:23.830 align:start position:0% information about events within your 00:02:23.830 --> 00:02:26.220 align:start position:0% information about events within your smart<00:02:24.100> contract<00:02:24.640> this<00:02:25.360> is<00:02:25.540> things<00:02:25.780> like<00:02:25.900> the 00:02:26.220 --> 00:02:26.230 align:start position:0% smart contract this is things like the 00:02:26.230 --> 00:02:28.650 align:start position:0% smart contract this is things like the users<00:02:26.500> token<00:02:26.890> balance<00:02:27.310> smart<00:02:28.150> contract 00:02:28.650 --> 00:02:28.660 align:start position:0% users token balance smart contract 00:02:28.660 --> 00:02:31.140 align:start position:0% users token balance smart contract function<00:02:29.020> executions<00:02:29.770> and<00:02:30.070> token<00:02:30.850> balance 00:02:31.140 --> 00:02:31.150 align:start position:0% function executions and token balance 00:02:31.150 --> 00:02:31.950 align:start position:0% function executions and token balance transfers 00:02:31.950 --> 00:02:31.960 align:start position:0% transfers 00:02:31.960 --> 00:02:34.530 align:start position:0% transfers however<00:02:32.950> contract<00:02:33.490> events<00:02:33.970> which<00:02:34.210> is<00:02:34.390> the 00:02:34.530 --> 00:02:34.540 align:start position:0% however contract events which is the 00:02:34.540 --> 00:02:36.300 align:start position:0% however contract events which is the state<00:02:34.870> of<00:02:35.020> your<00:02:35.170> smart<00:02:35.530> contract<00:02:36.100> are 00:02:36.300 --> 00:02:36.310 align:start position:0% state of your smart contract are 00:02:36.310 --> 00:02:39.270 align:start position:0% state of your smart contract are actually<00:02:37.300> implicit<00:02:37.660> and<00:02:37.960> computed<00:02:38.890> based<00:02:39.100> on 00:02:39.270 --> 00:02:39.280 align:start position:0% actually implicit and computed based on 00:02:39.280 --> 00:02:41.730 align:start position:0% actually implicit and computed based on the<00:02:39.370> data<00:02:39.550> in<00:02:39.760> the<00:02:39.940> blockchain<00:02:40.320> so<00:02:41.320> this<00:02:41.530> means 00:02:41.730 --> 00:02:41.740 align:start position:0% the data in the blockchain so this means 00:02:41.740 --> 00:02:43.680 align:start position:0% the data in the blockchain so this means that<00:02:41.920> your<00:02:42.070> DAP<00:02:42.280> needs<00:02:42.700> processing<00:02:43.480> power 00:02:43.680 --> 00:02:43.690 align:start position:0% that your DAP needs processing power 00:02:43.690 --> 00:02:46.170 align:start position:0% that your DAP needs processing power behind<00:02:44.080> it<00:02:44.530> when<00:02:45.490> you're<00:02:45.610> running<00:02:45.790> your 00:02:46.170 --> 00:02:46.180 align:start position:0% behind it when you're running your 00:02:46.180 --> 00:02:48.660 align:start position:0% behind it when you're running your ethereum<00:02:46.660> notes<00:02:47.100> if<00:02:48.100> you're<00:02:48.220> out<00:02:48.310> of<00:02:48.459> sync 00:02:48.660 --> 00:02:48.670 align:start position:0% ethereum notes if you're out of sync 00:02:48.670 --> 00:02:50.310 align:start position:0% ethereum notes if you're out of sync with<00:02:48.730> your<00:02:48.970> nodes<00:02:49.209> obviously<00:02:50.170> you're 00:02:50.310 --> 00:02:50.320 align:start position:0% with your nodes obviously you're 00:02:50.320 --> 00:02:51.750 align:start position:0% with your nodes obviously you're misrepresenting<00:02:50.530> the<00:02:51.130> state<00:02:51.340> of<00:02:51.459> the<00:02:51.550> world 00:02:51.750 --> 00:02:51.760 align:start position:0% misrepresenting the state of the world 00:02:51.760 --> 00:02:54.150 align:start position:0% misrepresenting the state of the world and<00:02:52.150> that's<00:02:52.600> a<00:02:52.750> dangerous<00:02:53.110> place<00:02:53.200> to<00:02:53.380> be<00:02:53.590> in<00:02:53.920> as 00:02:54.150 --> 00:02:54.160 align:start position:0% and that's a dangerous place to be in as 00:02:54.160 --> 00:02:59.010 align:start position:0% and that's a dangerous place to be in as an<00:02:54.700> application<00:02:57.209> so<00:02:58.209> with<00:02:58.540> the<00:02:58.630> importance<00:02:58.959> of 00:02:59.010 --> 00:02:59.020 align:start position:0% an application so with the importance of 00:02:59.020 --> 00:03:01.080 align:start position:0% an application so with the importance of staying<00:02:59.230> in<00:02:59.380> sync<00:02:59.800> now<00:03:00.040> for<00:03:00.370> your<00:03:00.489> DAP<00:03:00.730> kind<00:03:01.030> of 00:03:01.080 --> 00:03:01.090 align:start position:0% staying in sync now for your DAP kind of 00:03:01.090 --> 00:03:03.080 align:start position:0% staying in sync now for your DAP kind of resting<00:03:01.420> on<00:03:01.540> your<00:03:01.720> shoulders<00:03:02.140> as<00:03:02.320> a<00:03:02.350> developer 00:03:03.080 --> 00:03:03.090 align:start position:0% resting on your shoulders as a developer 00:03:03.090 --> 00:03:05.430 align:start position:0% resting on your shoulders as a developer some<00:03:04.090> of<00:03:04.180> you<00:03:04.270> guys<00:03:04.390> and<00:03:04.780> the<00:03:04.959> industry<00:03:05.320> as<00:03:05.410> a 00:03:05.430 --> 00:03:05.440 align:start position:0% some of you guys and the industry as a 00:03:05.440 --> 00:03:06.930 align:start position:0% some of you guys and the industry as a whole<00:03:05.530> we've<00:03:05.980> wondered<00:03:06.220> can<00:03:06.580> we<00:03:06.730> just 00:03:06.930 --> 00:03:06.940 align:start position:0% whole we've wondered can we just 00:03:06.940 --> 00:03:08.820 align:start position:0% whole we've wondered can we just outsource<00:03:07.420> this<00:03:07.690> and<00:03:07.900> pay<00:03:08.050> somebody<00:03:08.350> else<00:03:08.650> to 00:03:08.820 --> 00:03:08.830 align:start position:0% outsource this and pay somebody else to 00:03:08.830 --> 00:03:11.490 align:start position:0% outsource this and pay somebody else to do<00:03:08.980> it<00:03:09.160> and<00:03:09.570> the<00:03:10.570> answer<00:03:10.840> is<00:03:10.900> of<00:03:11.050> course<00:03:11.080> you 00:03:11.490 --> 00:03:11.500 align:start position:0% do it and the answer is of course you 00:03:11.500 --> 00:03:13.740 align:start position:0% do it and the answer is of course you can<00:03:11.820> there's<00:03:12.820> many<00:03:13.030> companies<00:03:13.270> out<00:03:13.690> there 00:03:13.740 --> 00:03:13.750 align:start position:0% can there's many companies out there 00:03:13.750 --> 00:03:16.140 align:start position:0% can there's many companies out there that<00:03:14.110> actually<00:03:14.230> offer<00:03:14.590> services<00:03:15.310> where<00:03:16.000> they 00:03:16.140 --> 00:03:16.150 align:start position:0% that actually offer services where they 00:03:16.150 --> 00:03:18.960 align:start position:0% that actually offer services where they will<00:03:16.300> run<00:03:16.540> nodes<00:03:16.900> for<00:03:17.290> you<00:03:17.500> not<00:03:18.400> all<00:03:18.640> of<00:03:18.820> them 00:03:18.960 --> 00:03:18.970 align:start position:0% will run nodes for you not all of them 00:03:18.970 --> 00:03:22.050 align:start position:0% will run nodes for you not all of them are<00:03:19.000> geared<00:03:19.630> towards<00:03:20.140> OLTP<00:03:20.950> processing<00:03:21.670> that 00:03:22.050 --> 00:03:22.060 align:start position:0% are geared towards OLTP processing that 00:03:22.060 --> 00:03:24.030 align:start position:0% are geared towards OLTP processing that adapt<00:03:22.450> needs<00:03:22.840> some<00:03:23.590> of<00:03:23.680> them<00:03:23.799> are<00:03:23.920> more 00:03:24.030 --> 00:03:24.040 align:start position:0% adapt needs some of them are more 00:03:24.040 --> 00:03:26.100 align:start position:0% adapt needs some of them are more analytics<00:03:24.670> based<00:03:24.910> you<00:03:25.299> know<00:03:25.330> Google<00:03:25.720> recently 00:03:26.100 --> 00:03:26.110 align:start position:0% analytics based you know Google recently 00:03:26.110 --> 00:03:28.920 align:start position:0% analytics based you know Google recently came<00:03:26.260> out<00:03:26.320> with<00:03:26.440> a<00:03:26.590> product<00:03:26.950> like<00:03:27.100> this<00:03:27.930> but 00:03:28.920 --> 00:03:28.930 align:start position:0% came out with a product like this but 00:03:28.930 --> 00:03:30.600 align:start position:0% came out with a product like this but the<00:03:29.140> question<00:03:29.739> we<00:03:29.830> need<00:03:29.890> to<00:03:29.950> ask<00:03:30.040> ourselves<00:03:30.280> as 00:03:30.600 --> 00:03:30.610 align:start position:0% the question we need to ask ourselves as 00:03:30.610 --> 00:03:33.240 align:start position:0% the question we need to ask ourselves as an<00:03:30.730> industry<00:03:30.970> is<00:03:31.360> do<00:03:32.350> we<00:03:32.590> want<00:03:32.890> to<00:03:33.100> move 00:03:33.240 --> 00:03:33.250 align:start position:0% an industry is do we want to move 00:03:33.250 --> 00:03:35.640 align:start position:0% an industry is do we want to move towards<00:03:33.549> the<00:03:33.760> MasterCard<00:03:34.269> Visa<00:03:34.450> model<00:03:34.959> right 00:03:35.640 --> 00:03:35.650 align:start position:0% towards the MasterCard Visa model right 00:03:35.650 --> 00:03:37.770 align:start position:0% towards the MasterCard Visa model right so<00:03:36.040> right<00:03:36.310> now<00:03:36.519> with<00:03:36.730> credit<00:03:37.090> card<00:03:37.269> processing 00:03:37.770 --> 00:03:37.780 align:start position:0% so right now with credit card processing 00:03:37.780 --> 00:03:39.390 align:start position:0% so right now with credit card processing there's<00:03:38.410> only<00:03:38.530> a<00:03:38.620> few<00:03:38.799> companies<00:03:39.100> in<00:03:39.220> the 00:03:39.390 --> 00:03:39.400 align:start position:0% there's only a few companies in the 00:03:39.400 --> 00:03:41.210 align:start position:0% there's only a few companies in the world<00:03:39.549> that<00:03:39.790> have<00:03:40.060> the<00:03:40.180> technical<00:03:40.510> expertise 00:03:41.210 --> 00:03:41.220 align:start position:0% world that have the technical expertise 00:03:41.220 --> 00:03:44.130 align:start position:0% world that have the technical expertise software<00:03:42.220> and<00:03:42.430> hardware<00:03:42.640> to<00:03:43.150> actually<00:03:43.930> keep 00:03:44.130 --> 00:03:44.140 align:start position:0% software and hardware to actually keep 00:03:44.140 --> 00:03:46.020 align:start position:0% software and hardware to actually keep up<00:03:44.350> with<00:03:44.530> the<00:03:44.680> number<00:03:45.040> of<00:03:45.280> transactions<00:03:45.910> that 00:03:46.020 --> 00:03:46.030 align:start position:0% up with the number of transactions that 00:03:46.030 --> 00:03:49.110 align:start position:0% up with the number of transactions that happen<00:03:46.360> on<00:03:46.450> credit<00:03:46.720> cards<00:03:46.959> and<00:03:47.230> as<00:03:48.190> we<00:03:48.610> move 00:03:49.110 --> 00:03:49.120 align:start position:0% happen on credit cards and as we move 00:03:49.120 --> 00:03:51.090 align:start position:0% happen on credit cards and as we move state<00:03:49.660> management<00:03:50.110> towards<00:03:50.620> centralized 00:03:51.090 --> 00:03:51.100 align:start position:0% state management towards centralized 00:03:51.100 --> 00:03:53.580 align:start position:0% state management towards centralized parties<00:03:51.610> we<00:03:52.570> move<00:03:52.810> the<00:03:52.959> entire<00:03:53.260> industry 00:03:53.580 --> 00:03:53.590 align:start position:0% parties we move the entire industry 00:03:53.590 --> 00:03:55.770 align:start position:0% parties we move the entire industry towards<00:03:54.070> that<00:03:54.250> model<00:03:54.610> again<00:03:54.880> and<00:03:55.120> the 00:03:55.770 --> 00:03:55.780 align:start position:0% towards that model again and the 00:03:55.780 --> 00:03:57.990 align:start position:0% towards that model again and the question<00:03:56.140> is<00:03:56.290> do<00:03:56.769> we<00:03:56.830> want<00:03:56.950> to<00:03:57.310> do<00:03:57.459> that<00:03:57.610> do<00:03:57.880> we 00:03:57.990 --> 00:03:58.000 align:start position:0% question is do we want to do that do we 00:03:58.000 --> 00:04:02.820 align:start position:0% question is do we want to do that do we actually<00:03:58.330> want<00:03:58.480> to<00:03:58.510> be<00:03:58.600> there<00:04:00.690> the<00:04:01.830> solution 00:04:02.820 --> 00:04:02.830 align:start position:0% actually want to be there the solution 00:04:02.830 --> 00:04:05.280 align:start position:0% actually want to be there the solution in<00:04:03.130> our<00:04:03.580> mind<00:04:03.850> to<00:04:04.030> combat<00:04:04.450> the<00:04:04.570> centralization 00:04:05.280 --> 00:04:05.290 align:start position:0% in our mind to combat the centralization 00:04:05.290 --> 00:04:09.360 align:start position:0% in our mind to combat the centralization is<00:04:05.500> having<00:04:06.340> people<00:04:06.549> run<00:04:07.420> their<00:04:07.630> own<00:04:07.660> nodes<00:04:08.370> but 00:04:09.360 --> 00:04:09.370 align:start position:0% is having people run their own nodes but 00:04:09.370 --> 00:04:12.470 align:start position:0% is having people run their own nodes but running<00:04:09.970> an<00:04:10.180> ethereum<00:04:10.540> node<00:04:10.780> at<00:04:11.110> scale<00:04:11.950> in 00:04:12.470 --> 00:04:12.480 align:start position:0% running an ethereum node at scale in 00:04:12.480 --> 00:04:15.390 align:start position:0% running an ethereum node at scale in production<00:04:13.480> is<00:04:13.750> really<00:04:14.500> not<00:04:14.650> well<00:04:14.890> documented 00:04:15.390 --> 00:04:15.400 align:start position:0% production is really not well documented 00:04:15.400 --> 00:04:19.560 align:start position:0% production is really not well documented online<00:04:15.970> right<00:04:16.660> now<00:04:17.549> there's<00:04:18.549> also<00:04:18.910> costs 00:04:19.560 --> 00:04:19.570 align:start position:0% online right now there's also costs 00:04:19.570 --> 00:04:21.740 align:start position:0% online right now there's also costs associated<00:04:20.320> with<00:04:20.560> running<00:04:20.709> your<00:04:20.979> own<00:04:21.010> notes 00:04:21.740 --> 00:04:21.750 align:start position:0% associated with running your own notes 00:04:21.750 --> 00:04:25.500 align:start position:0% associated with running your own notes not<00:04:22.750> well-documented<00:04:23.320> either<00:04:24.120> so<00:04:25.120> not<00:04:25.390> only 00:04:25.500 --> 00:04:25.510 align:start position:0% not well-documented either so not only 00:04:25.510 --> 00:04:27.090 align:start position:0% not well-documented either so not only do<00:04:25.660> you<00:04:25.720> have<00:04:25.780> the<00:04:25.960> overhead<00:04:26.320> costs<00:04:26.770> of 00:04:27.090 --> 00:04:27.100 align:start position:0% do you have the overhead costs of 00:04:27.100 --> 00:04:28.710 align:start position:0% do you have the overhead costs of the<00:04:27.160> actual<00:04:27.550> infrastructure<00:04:28.090> for<00:04:28.390> running 00:04:28.710 --> 00:04:28.720 align:start position:0% the actual infrastructure for running 00:04:28.720 --> 00:04:31.200 align:start position:0% the actual infrastructure for running your<00:04:28.870> own<00:04:28.900> node<00:04:29.290> but<00:04:30.040> you<00:04:30.130> also<00:04:30.310> have<00:04:30.580> costs<00:04:31.060> on 00:04:31.200 --> 00:04:31.210 align:start position:0% your own node but you also have costs on 00:04:31.210 --> 00:04:33.300 align:start position:0% your own node but you also have costs on your<00:04:31.360> team<00:04:31.660> to<00:04:31.990> create<00:04:32.260> the<00:04:32.620> infrastructure 00:04:33.300 --> 00:04:33.310 align:start position:0% your team to create the infrastructure 00:04:33.310 --> 00:04:36.360 align:start position:0% your team to create the infrastructure and<00:04:33.520> create<00:04:34.270> the<00:04:34.570> software<00:04:35.410> that<00:04:36.130> manages 00:04:36.360 --> 00:04:36.370 align:start position:0% and create the software that manages 00:04:36.370 --> 00:04:38.070 align:start position:0% and create the software that manages your<00:04:36.700> nodes<00:04:37.030> make<00:04:37.330> sure<00:04:37.540> that<00:04:37.720> your<00:04:37.900> nodes 00:04:38.070 --> 00:04:38.080 align:start position:0% your nodes make sure that your nodes 00:04:38.080 --> 00:04:41.790 align:start position:0% your nodes make sure that your nodes stay<00:04:38.350> in<00:04:38.500> sync<00:04:39.480> there's<00:04:40.480> a<00:04:40.720> lot<00:04:41.110> of<00:04:41.260> resources 00:04:41.790 --> 00:04:41.800 align:start position:0% stay in sync there's a lot of resources 00:04:41.800 --> 00:04:43.620 align:start position:0% stay in sync there's a lot of resources online<00:04:41.980> pertaining<00:04:42.580> to<00:04:42.700> optimizing<00:04:43.240> nodes 00:04:43.620 --> 00:04:43.630 align:start position:0% online pertaining to optimizing nodes 00:04:43.630 --> 00:04:46.140 align:start position:0% online pertaining to optimizing nodes for<00:04:44.020> mining<00:04:44.700> but<00:04:45.700> there's<00:04:45.850> not<00:04:46.000> many 00:04:46.140 --> 00:04:46.150 align:start position:0% for mining but there's not many 00:04:46.150 --> 00:04:50.490 align:start position:0% for mining but there's not many resources<00:04:46.950> optimized<00:04:47.950> for<00:04:48.940> reading<00:04:49.510> and<00:04:49.810> read 00:04:50.490 --> 00:04:50.500 align:start position:0% resources optimized for reading and read 00:04:50.500 --> 00:04:55.350 align:start position:0% resources optimized for reading and read heavy<00:04:50.800> workloads<00:04:51.430> that<00:04:52.000> adapt<00:04:52.540> needs<00:04:54.060> it<00:04:55.060> gets 00:04:55.350 --> 00:04:55.360 align:start position:0% heavy workloads that adapt needs it gets 00:04:55.360 --> 00:04:56.820 align:start position:0% heavy workloads that adapt needs it gets even<00:04:55.630> worse<00:04:55.660> when<00:04:56.020> you<00:04:56.200> start<00:04:56.410> going<00:04:56.530> into 00:04:56.820 --> 00:04:56.830 align:start position:0% even worse when you start going into 00:04:56.830 --> 00:04:59.550 align:start position:0% even worse when you start going into data<00:04:57.130> querying<00:04:57.730> and<00:04:58.150> trying<00:04:58.780> to<00:04:58.960> understand 00:04:59.550 --> 00:04:59.560 align:start position:0% data querying and trying to understand 00:04:59.560 --> 00:05:02.700 align:start position:0% data querying and trying to understand how<00:04:59.650> you<00:04:59.830> process<00:05:00.310> this<00:05:00.790> data<00:05:01.320> there's<00:05:02.320> very 00:05:02.700 --> 00:05:02.710 align:start position:0% how you process this data there's very 00:05:02.710 --> 00:05:05.040 align:start position:0% how you process this data there's very few<00:05:02.830> resources<00:05:03.760> online<00:05:03.940> for<00:05:04.180> data<00:05:04.570> querying 00:05:05.040 --> 00:05:05.050 align:start position:0% few resources online for data querying 00:05:05.050 --> 00:05:07.470 align:start position:0% few resources online for data querying for<00:05:05.170> taking<00:05:05.860> block<00:05:06.280> data<00:05:06.640> and<00:05:06.940> storing<00:05:07.360> it 00:05:07.470 --> 00:05:07.480 align:start position:0% for taking block data and storing it 00:05:07.480 --> 00:05:10.380 align:start position:0% for taking block data and storing it internally<00:05:08.310> we<00:05:09.310> as<00:05:09.670> well<00:05:09.940> as<00:05:10.030> everyone<00:05:10.210> else 00:05:10.380 --> 00:05:10.390 align:start position:0% internally we as well as everyone else 00:05:10.390 --> 00:05:11.820 align:start position:0% internally we as well as everyone else who<00:05:10.450> was<00:05:10.480> up<00:05:10.660> here<00:05:10.810> before<00:05:11.140> I<00:05:11.170> have<00:05:11.320> created 00:05:11.820 --> 00:05:11.830 align:start position:0% who was up here before I have created 00:05:11.830 --> 00:05:13.620 align:start position:0% who was up here before I have created tools<00:05:12.040> for<00:05:12.370> doing<00:05:12.490> that 00:05:13.620 --> 00:05:13.630 align:start position:0% tools for doing that 00:05:13.630 --> 00:05:16.560 align:start position:0% tools for doing that we're<00:05:14.320> not<00:05:14.410> outsourcing<00:05:15.010> those<00:05:15.160> yet<00:05:15.400> but<00:05:16.240> you 00:05:16.560 --> 00:05:16.570 align:start position:0% we're not outsourcing those yet but you 00:05:16.570 --> 00:05:17.880 align:start position:0% we're not outsourcing those yet but you know<00:05:16.630> that<00:05:16.810> is<00:05:16.930> something<00:05:17.260> that<00:05:17.410> we<00:05:17.500> are<00:05:17.530> all 00:05:17.880 --> 00:05:17.890 align:start position:0% know that is something that we are all 00:05:17.890 --> 00:05:20.220 align:start position:0% know that is something that we are all creating<00:05:18.400> our<00:05:18.610> own<00:05:18.760> data<00:05:19.120> querying<00:05:19.660> layer<00:05:19.960> on 00:05:20.220 --> 00:05:20.230 align:start position:0% creating our own data querying layer on 00:05:20.230 --> 00:05:22.650 align:start position:0% creating our own data querying layer on top<00:05:20.500> of<00:05:20.710> the<00:05:20.800> blockchain<00:05:21.040> and<00:05:21.730> there<00:05:22.360> are<00:05:22.450> some 00:05:22.650 --> 00:05:22.660 align:start position:0% top of the blockchain and there are some 00:05:22.660 --> 00:05:23.850 align:start position:0% top of the blockchain and there are some companies<00:05:22.900> right<00:05:23.170> now<00:05:23.290> like<00:05:23.500> the<00:05:23.620> graph<00:05:23.830> that 00:05:23.850 --> 00:05:23.860 align:start position:0% companies right now like the graph that 00:05:23.860 --> 00:05:26.610 align:start position:0% companies right now like the graph that are<00:05:24.190> actively<00:05:24.460> working<00:05:24.580> on<00:05:25.180> making<00:05:26.110> a<00:05:26.380> way<00:05:26.560> to 00:05:26.610 --> 00:05:26.620 align:start position:0% are actively working on making a way to 00:05:26.620 --> 00:05:31.500 align:start position:0% are actively working on making a way to query<00:05:27.600> using<00:05:28.600> graph<00:05:28.870> QL<00:05:29.140> but<00:05:29.860> they<00:05:30.760> also<00:05:31.060> still 00:05:31.500 --> 00:05:31.510 align:start position:0% query using graph QL but they also still 00:05:31.510 --> 00:05:33.570 align:start position:0% query using graph QL but they also still require<00:05:31.810> that<00:05:31.960> you<00:05:32.080> run<00:05:32.290> your<00:05:32.470> own<00:05:32.500> nodes<00:05:32.920> or 00:05:33.570 --> 00:05:33.580 align:start position:0% require that you run your own nodes or 00:05:33.580 --> 00:05:36.000 align:start position:0% require that you run your own nodes or you<00:05:33.730> find<00:05:34.000> somebody<00:05:34.450> that<00:05:34.840> can<00:05:35.080> run<00:05:35.260> nodes<00:05:35.590> for 00:05:36.000 --> 00:05:36.010 align:start position:0% you find somebody that can run nodes for 00:05:36.010 --> 00:05:40.920 align:start position:0% you find somebody that can run nodes for you<00:05:36.250> to<00:05:36.760> get<00:05:36.880> the<00:05:36.970> underlying<00:05:37.300> state<00:05:37.690> data<00:05:39.930> so 00:05:40.920 --> 00:05:40.930 align:start position:0% you to get the underlying state data so 00:05:40.930 --> 00:05:43.560 align:start position:0% you to get the underlying state data so right<00:05:41.110> here<00:05:41.410> is<00:05:41.650> a<00:05:42.130> simplistic<00:05:42.670> view<00:05:42.820> of<00:05:43.240> our 00:05:43.560 --> 00:05:43.570 align:start position:0% right here is a simplistic view of our 00:05:43.570 --> 00:05:48.210 align:start position:0% right here is a simplistic view of our death<00:05:43.810> architecture<00:05:45.480> we<00:05:46.480> run<00:05:47.020> a<00:05:47.410> production 00:05:48.210 --> 00:05:48.220 align:start position:0% death architecture we run a production 00:05:48.220 --> 00:05:51.620 align:start position:0% death architecture we run a production cluster<00:05:48.730> and<00:05:49.240> it's<00:05:49.960> basically<00:05:50.530> a<00:05:50.920> group<00:05:51.280> of 00:05:51.620 --> 00:05:51.630 align:start position:0% cluster and it's basically a group of 00:05:51.630 --> 00:05:54.990 align:start position:0% cluster and it's basically a group of load-balanced<00:05:52.630> death<00:05:52.960> nodes<00:05:53.290> behind<00:05:54.220> RS<00:05:54.850> re 00:05:54.990 --> 00:05:55.000 align:start position:0% load-balanced death nodes behind RS re 00:05:55.000 --> 00:05:57.090 align:start position:0% load-balanced death nodes behind RS re in<00:05:55.180> an<00:05:55.300> auto<00:05:55.450> scaling<00:05:55.810> group<00:05:55.980> behind<00:05:56.980> an 00:05:57.090 --> 00:05:57.100 align:start position:0% in an auto scaling group behind an 00:05:57.100 --> 00:06:01.290 align:start position:0% in an auto scaling group behind an application<00:05:57.610> load<00:05:57.790> balancer<00:05:58.330> on<00:05:58.510> AWS<00:06:00.300> the 00:06:01.290 --> 00:06:01.300 align:start position:0% application load balancer on AWS the 00:06:01.300 --> 00:06:03.300 align:start position:0% application load balancer on AWS the application<00:06:02.080> load<00:06:02.230> balancer<00:06:02.620> allows<00:06:02.860> us<00:06:03.070> to 00:06:03.300 --> 00:06:03.310 align:start position:0% application load balancer allows us to 00:06:03.310 --> 00:06:07.200 align:start position:0% application load balancer allows us to upgrade<00:06:03.520> non-secure<00:06:04.300> requests<00:06:04.810> to<00:06:05.350> TLS<00:06:06.210> we 00:06:07.200 --> 00:06:07.210 align:start position:0% upgrade non-secure requests to TLS we 00:06:07.210 --> 00:06:10.560 align:start position:0% upgrade non-secure requests to TLS we can<00:06:07.360> also<00:06:07.510> run<00:06:07.960> very<00:06:08.590> tight<00:06:08.950> health<00:06:09.220> checks<00:06:09.580> to 00:06:10.560 --> 00:06:10.570 align:start position:0% can also run very tight health checks to 00:06:10.570 --> 00:06:13.440 align:start position:0% can also run very tight health checks to cycle<00:06:10.960> out<00:06:11.020> nodes<00:06:11.380> that<00:06:11.650> are<00:06:11.710> not<00:06:11.830> healthy<00:06:12.610> out 00:06:13.440 --> 00:06:13.450 align:start position:0% cycle out nodes that are not healthy out 00:06:13.450 --> 00:06:15.750 align:start position:0% cycle out nodes that are not healthy out of<00:06:13.600> the<00:06:13.690> load<00:06:13.840> balancer<00:06:14.320> target<00:06:14.680> group<00:06:14.860> so<00:06:15.640> if 00:06:15.750 --> 00:06:15.760 align:start position:0% of the load balancer target group so if 00:06:15.760 --> 00:06:17.010 align:start position:0% of the load balancer target group so if they<00:06:15.910> fall<00:06:16.090> behind<00:06:16.240> for<00:06:16.600> instance<00:06:16.810> like<00:06:16.900> the 00:06:17.010 --> 00:06:17.020 align:start position:0% they fall behind for instance like the 00:06:17.020 --> 00:06:19.830 align:start position:0% they fall behind for instance like the head<00:06:17.140> of<00:06:17.260> the<00:06:17.320> chain<00:06:17.560> and<00:06:18.420> usually<00:06:19.420> what<00:06:19.720> we 00:06:19.830 --> 00:06:19.840 align:start position:0% head of the chain and usually what we 00:06:19.840 --> 00:06:21.870 align:start position:0% head of the chain and usually what we found<00:06:20.110> in<00:06:20.290> our<00:06:20.380> internal<00:06:20.770> test<00:06:21.580> is<00:06:21.760> we 00:06:21.870 --> 00:06:21.880 align:start position:0% found in our internal test is we 00:06:21.880 --> 00:06:23.220 align:start position:0% found in our internal test is we recommend<00:06:22.270> studying<00:06:22.420> the<00:06:22.600> frequency<00:06:23.140> of 00:06:23.220 --> 00:06:23.230 align:start position:0% recommend studying the frequency of 00:06:23.230 --> 00:06:25.290 align:start position:0% recommend studying the frequency of those<00:06:23.350> health<00:06:23.590> checks<00:06:23.920> to<00:06:24.670> a<00:06:24.700> time<00:06:25.030> that<00:06:25.210> is 00:06:25.290 --> 00:06:25.300 align:start position:0% those health checks to a time that is 00:06:25.300 --> 00:06:27.780 align:start position:0% those health checks to a time that is less<00:06:25.510> than<00:06:25.780> a<00:06:25.900> single<00:06:26.350> block<00:06:26.790> propagation 00:06:27.780 --> 00:06:27.790 align:start position:0% less than a single block propagation 00:06:27.790 --> 00:06:29.850 align:start position:0% less than a single block propagation time<00:06:28.000> so<00:06:28.630> right<00:06:28.930> now<00:06:29.140> that's<00:06:29.320> doing<00:06:29.590> a<00:06:29.680> health 00:06:29.850 --> 00:06:29.860 align:start position:0% time so right now that's doing a health 00:06:29.860 --> 00:06:33.960 align:start position:0% time so right now that's doing a health check<00:06:30.270> inside<00:06:31.270> of<00:06:31.420> a<00:06:31.480> 15-second<00:06:32.260> window<00:06:32.970> this 00:06:33.960 --> 00:06:33.970 align:start position:0% check inside of a 15-second window this 00:06:33.970 --> 00:06:36.300 align:start position:0% check inside of a 15-second window this allows<00:06:34.270> nodes<00:06:34.510> that<00:06:34.630> fall<00:06:34.870> behind<00:06:35.050> to<00:06:36.040> be 00:06:36.300 --> 00:06:36.310 align:start position:0% allows nodes that fall behind to be 00:06:36.310 --> 00:06:37.680 align:start position:0% allows nodes that fall behind to be dropped<00:06:36.550> at<00:06:36.700> the<00:06:36.790> pool<00:06:36.970> quickly<00:06:37.360> so<00:06:37.570> they<00:06:37.660> can 00:06:37.680 --> 00:06:37.690 align:start position:0% dropped at the pool quickly so they can 00:06:37.690 --> 00:06:39.900 align:start position:0% dropped at the pool quickly so they can sink<00:06:38.140> back<00:06:38.380> in<00:06:38.710> so<00:06:39.250> you're<00:06:39.400> not<00:06:39.430> querying 00:06:39.900 --> 00:06:39.910 align:start position:0% sink back in so you're not querying 00:06:39.910 --> 00:06:40.500 align:start position:0% sink back in so you're not querying nodes 00:06:40.500 --> 00:06:40.510 align:start position:0% nodes 00:06:40.510 --> 00:06:43.620 align:start position:0% nodes that<00:06:40.960> you<00:06:41.080> know<00:06:41.290> are<00:06:41.620> behind<00:06:41.980> in<00:06:42.130> state<00:06:42.630> the 00:06:43.620 --> 00:06:43.630 align:start position:0% that you know are behind in state the 00:06:43.630 --> 00:06:45.690 align:start position:0% that you know are behind in state the load-balanced<00:06:44.320> nature<00:06:44.620> of<00:06:44.710> our<00:06:44.830> clusters 00:06:45.690 --> 00:06:45.700 align:start position:0% load-balanced nature of our clusters 00:06:45.700 --> 00:06:47.310 align:start position:0% load-balanced nature of our clusters also<00:06:46.060> means<00:06:46.240> that<00:06:46.330> if<00:06:46.540> a<00:06:46.630> single<00:06:46.870> node<00:06:47.080> does 00:06:47.310 --> 00:06:47.320 align:start position:0% also means that if a single node does 00:06:47.320 --> 00:06:49.500 align:start position:0% also means that if a single node does fall<00:06:47.560> behind<00:06:47.860> our<00:06:48.340> overarching<00:06:48.820> service 00:06:49.500 --> 00:06:49.510 align:start position:0% fall behind our overarching service 00:06:49.510 --> 00:06:51.840 align:start position:0% fall behind our overarching service doesn't<00:06:50.350> actually<00:06:50.560> fall<00:06:51.070> out<00:06:51.130> of<00:06:51.370> sync<00:06:51.670> and 00:06:51.840 --> 00:06:51.850 align:start position:0% doesn't actually fall out of sync and 00:06:51.850 --> 00:06:57.330 align:start position:0% doesn't actually fall out of sync and become<00:06:52.120> unavailable<00:06:55.410> so<00:06:56.410> we<00:06:56.680> use<00:06:56.890> our<00:06:57.100> goth 00:06:57.330 --> 00:06:57.340 align:start position:0% become unavailable so we use our goth 00:06:57.340 --> 00:06:59.520 align:start position:0% become unavailable so we use our goth nodes<00:06:57.610> on<00:06:57.700> the<00:06:57.790> front<00:06:58.030> end<00:06:58.210> you<00:06:58.300> I<00:06:58.480> on<00:06:59.290> the<00:06:59.440> air 00:06:59.520 --> 00:06:59.530 align:start position:0% nodes on the front end you I on the air 00:06:59.530 --> 00:07:02.520 align:start position:0% nodes on the front end you I on the air swap<00:06:59.800> platform<00:07:00.340> as<00:07:00.660> well<00:07:01.660> as<00:07:01.720> in<00:07:02.230> the<00:07:02.320> backend 00:07:02.520 --> 00:07:02.530 align:start position:0% swap platform as well as in the backend 00:07:02.530 --> 00:07:05.790 align:start position:0% swap platform as well as in the backend for<00:07:03.160> state<00:07:03.340> management<00:07:03.700> and<00:07:04.030> storage<00:07:04.650> on<00:07:05.650> the 00:07:05.790 --> 00:07:05.800 align:start position:0% for state management and storage on the 00:07:05.800 --> 00:07:08.160 align:start position:0% for state management and storage on the front<00:07:06.040> end<00:07:06.220> this<00:07:06.370> takes<00:07:06.610> the<00:07:06.700> form<00:07:06.820> of<00:07:07.300> balance 00:07:08.160 --> 00:07:08.170 align:start position:0% front end this takes the form of balance 00:07:08.170 --> 00:07:10.710 align:start position:0% front end this takes the form of balance checks<00:07:08.680> in<00:07:08.980> the<00:07:09.100> UI<00:07:09.480> wrapping<00:07:10.480> and<00:07:10.630> unwrapping 00:07:10.710 --> 00:07:10.720 align:start position:0% checks in the UI wrapping and unwrapping 00:07:10.720 --> 00:07:14.130 align:start position:0% checks in the UI wrapping and unwrapping width<00:07:11.410> and<00:07:11.800> also<00:07:12.790> approvals<00:07:13.510> an<00:07:13.750> approval 00:07:14.130 --> 00:07:14.140 align:start position:0% width and also approvals an approval 00:07:14.140 --> 00:07:16.080 align:start position:0% width and also approvals an approval state<00:07:14.350> for<00:07:14.530> the<00:07:14.590> atomic<00:07:14.800> swaps<00:07:15.220> so<00:07:15.850> if<00:07:15.940> you've 00:07:16.080 --> 00:07:16.090 align:start position:0% state for the atomic swaps so if you've 00:07:16.090 --> 00:07:19.140 align:start position:0% state for the atomic swaps so if you've used<00:07:16.150> the<00:07:17.080> air<00:07:17.290> swap<00:07:17.530> platform<00:07:18.070> you<00:07:18.760> know<00:07:18.910> that 00:07:19.140 --> 00:07:19.150 align:start position:0% used the air swap platform you know that 00:07:19.150 --> 00:07:21.240 align:start position:0% used the air swap platform you know that I<00:07:19.180> like<00:07:19.390> on<00:07:19.540> the<00:07:19.660> new<00:07:19.860> spaces<00:07:20.860> on<00:07:21.010> the<00:07:21.100> right 00:07:21.240 --> 00:07:21.250 align:start position:0% I like on the new spaces on the right 00:07:21.250 --> 00:07:22.800 align:start position:0% I like on the new spaces on the right hand<00:07:21.460> side<00:07:21.670> you<00:07:21.790> know<00:07:21.850> we<00:07:21.970> have<00:07:22.180> a<00:07:22.480> bunch<00:07:22.720> of 00:07:22.800 --> 00:07:22.810 align:start position:0% hand side you know we have a bunch of 00:07:22.810 --> 00:07:24.810 align:start position:0% hand side you know we have a bunch of information<00:07:23.380> that<00:07:23.830> comes<00:07:24.190> from<00:07:24.310> our<00:07:24.430> Gotthard 00:07:24.810 --> 00:07:24.820 align:start position:0% information that comes from our Gotthard 00:07:24.820 --> 00:07:27.840 align:start position:0% information that comes from our Gotthard state<00:07:25.710> we<00:07:26.710> also<00:07:26.890> use<00:07:27.100> a<00:07:27.130> heavy<00:07:27.310> polling<00:07:27.730> method 00:07:27.840 --> 00:07:27.850 align:start position:0% state we also use a heavy polling method 00:07:27.850 --> 00:07:30.810 align:start position:0% state we also use a heavy polling method to<00:07:28.330> handle<00:07:28.630> trade<00:07:29.080> confirmations<00:07:29.680> so<00:07:30.550> if 00:07:30.810 --> 00:07:30.820 align:start position:0% to handle trade confirmations so if 00:07:30.820 --> 00:07:32.130 align:start position:0% to handle trade confirmations so if you've<00:07:30.940> ever<00:07:31.090> popped<00:07:31.360> open<00:07:31.510> the<00:07:31.720> network<00:07:31.870> tab 00:07:32.130 --> 00:07:32.140 align:start position:0% you've ever popped open the network tab 00:07:32.140 --> 00:07:34.740 align:start position:0% you've ever popped open the network tab in<00:07:32.320> chrome<00:07:32.350> while<00:07:33.040> using<00:07:33.310> air<00:07:33.640> swap<00:07:33.940> you'll 00:07:34.740 --> 00:07:34.750 align:start position:0% in chrome while using air swap you'll 00:07:34.750 --> 00:07:36.300 align:start position:0% in chrome while using air swap you'll notice<00:07:35.050> that<00:07:35.170> we<00:07:35.290> do<00:07:35.410> heavy<00:07:35.590> polling<00:07:36.040> against 00:07:36.300 --> 00:07:36.310 align:start position:0% notice that we do heavy polling against 00:07:36.310 --> 00:07:39.360 align:start position:0% notice that we do heavy polling against a<00:07:36.490> gap<00:07:36.700> cluster<00:07:37.410> basically<00:07:38.410> when<00:07:38.680> you<00:07:38.890> do<00:07:39.340> a 00:07:39.360 --> 00:07:39.370 align:start position:0% a gap cluster basically when you do a 00:07:39.370 --> 00:07:41.130 align:start position:0% a gap cluster basically when you do a trade<00:07:39.820> to<00:07:40.030> make<00:07:40.120> sure<00:07:40.300> the<00:07:40.480> trade<00:07:40.690> actually 00:07:41.130 --> 00:07:41.140 align:start position:0% trade to make sure the trade actually 00:07:41.140 --> 00:07:43.800 align:start position:0% trade to make sure the trade actually goes<00:07:41.260> through<00:07:41.470> and<00:07:41.560> gets<00:07:41.680> executed<00:07:42.660> we<00:07:43.660> also 00:07:43.800 --> 00:07:43.810 align:start position:0% goes through and gets executed we also 00:07:43.810 --> 00:07:46.050 align:start position:0% goes through and gets executed we also do<00:07:44.020> allow<00:07:44.230> limited<00:07:44.560> very<00:07:45.310> limited<00:07:45.790> third 00:07:46.050 --> 00:07:46.060 align:start position:0% do allow limited very limited third 00:07:46.060 --> 00:07:49.470 align:start position:0% do allow limited very limited third party<00:07:46.450> access<00:07:46.900> to<00:07:47.260> our<00:07:47.500> GAF<00:07:47.710> cluster<00:07:48.360> we<00:07:49.360> do 00:07:49.470 --> 00:07:49.480 align:start position:0% party access to our GAF cluster we do 00:07:49.480 --> 00:07:52.080 align:start position:0% party access to our GAF cluster we do have<00:07:49.720> some<00:07:50.080> of<00:07:50.170> our<00:07:50.260> maker<00:07:50.440> partners<00:07:51.010> who<00:07:51.760> use 00:07:52.080 --> 00:07:52.090 align:start position:0% have some of our maker partners who use 00:07:52.090 --> 00:07:54.360 align:start position:0% have some of our maker partners who use our<00:07:52.390> GAF<00:07:52.690> cluster<00:07:53.170> as<00:07:53.320> well<00:07:53.530> for<00:07:53.920> balance 00:07:54.360 --> 00:07:54.370 align:start position:0% our GAF cluster as well for balance 00:07:54.370 --> 00:07:57.630 align:start position:0% our GAF cluster as well for balance checks<00:07:54.700> both<00:07:55.480> for<00:07:55.870> their<00:07:56.140> own<00:07:56.320> wallets<00:07:56.830> that 00:07:57.630 --> 00:07:57.640 align:start position:0% checks both for their own wallets that 00:07:57.640 --> 00:07:59.130 align:start position:0% checks both for their own wallets that they<00:07:57.730> operate<00:07:58.090> out<00:07:58.270> of<00:07:58.420> as<00:07:58.600> well<00:07:58.900> as 00:07:59.130 --> 00:07:59.140 align:start position:0% they operate out of as well as 00:07:59.140 --> 00:08:00.930 align:start position:0% they operate out of as well as counterparty<00:07:59.830> balance<00:08:00.130> checks<00:08:00.400> to<00:08:00.610> make<00:08:00.730> sure 00:08:00.930 --> 00:08:00.940 align:start position:0% counterparty balance checks to make sure 00:08:00.940 --> 00:08:02.550 align:start position:0% counterparty balance checks to make sure that<00:08:01.060> the<00:08:01.180> counterparty<00:08:01.690> taking<00:08:01.990> a<00:08:02.080> trade<00:08:02.350> is 00:08:02.550 --> 00:08:02.560 align:start position:0% that the counterparty taking a trade is 00:08:02.560 --> 00:08:06.750 align:start position:0% that the counterparty taking a trade is a<00:08:03.430> valid<00:08:04.180> taker<00:08:05.250> further<00:08:06.250> our<00:08:06.430> parties<00:08:06.730> and 00:08:06.750 --> 00:08:06.760 align:start position:0% a valid taker further our parties and 00:08:06.760 --> 00:08:09.390 align:start position:0% a valid taker further our parties and also<00:08:07.090> use<00:08:07.330> the<00:08:07.750> GAF<00:08:08.140> cluster<00:08:08.560> for<00:08:08.800> market<00:08:09.250> data 00:08:09.390 --> 00:08:09.400 align:start position:0% also use the GAF cluster for market data 00:08:09.400 --> 00:08:12.390 align:start position:0% also use the GAF cluster for market data and<00:08:09.550> price<00:08:10.060> information<00:08:10.920> in<00:08:11.920> addition<00:08:12.130> with<00:08:12.370> a 00:08:12.390 --> 00:08:12.400 align:start position:0% and price information in addition with a 00:08:12.400 --> 00:08:14.340 align:start position:0% and price information in addition with a couple<00:08:12.610> of<00:08:12.670> their<00:08:12.790> sources<00:08:13.120> but<00:08:13.750> we<00:08:13.930> we<00:08:14.200> do 00:08:14.340 --> 00:08:14.350 align:start position:0% couple of their sources but we we do 00:08:14.350 --> 00:08:17.130 align:start position:0% couple of their sources but we we do provide<00:08:15.040> chain<00:08:15.400> state<00:08:15.700> to<00:08:15.820> them<00:08:15.970> for<00:08:16.330> pricing 00:08:17.130 --> 00:08:17.140 align:start position:0% provide chain state to them for pricing 00:08:17.140 --> 00:08:21.210 align:start position:0% provide chain state to them for pricing information<00:08:19.590> when<00:08:20.590> we<00:08:20.710> were<00:08:20.830> setting<00:08:20.980> up<00:08:21.100> our 00:08:21.210 --> 00:08:21.220 align:start position:0% information when we were setting up our 00:08:21.220 --> 00:08:23.070 align:start position:0% information when we were setting up our Gotha<00:08:21.430> cluster<00:08:21.700> for<00:08:21.850> the<00:08:21.910> first<00:08:22.120> time<00:08:22.420> we<00:08:22.930> ran 00:08:23.070 --> 00:08:23.080 align:start position:0% Gotha cluster for the first time we ran 00:08:23.080 --> 00:08:25.470 align:start position:0% Gotha cluster for the first time we ran into<00:08:23.230> many<00:08:23.440> issues<00:08:23.650> with<00:08:24.640> the<00:08:24.850> way<00:08:24.970> gift<00:08:25.270> was 00:08:25.470 --> 00:08:25.480 align:start position:0% into many issues with the way gift was 00:08:25.480 --> 00:08:27.660 align:start position:0% into many issues with the way gift was written<00:08:25.690> and<00:08:25.990> our<00:08:26.860> particular<00:08:27.220> use<00:08:27.400> case<00:08:27.640> of 00:08:27.660 --> 00:08:27.670 align:start position:0% written and our particular use case of 00:08:27.670 --> 00:08:29.970 align:start position:0% written and our particular use case of heavy<00:08:27.970> reads<00:08:28.290> this<00:08:29.290> is<00:08:29.410> not<00:08:29.530> a<00:08:29.560> knock<00:08:29.770> and<00:08:29.890> the 00:08:29.970 --> 00:08:29.980 align:start position:0% heavy reads this is not a knock and the 00:08:29.980 --> 00:08:31.260 align:start position:0% heavy reads this is not a knock and the guys<00:08:30.130> that<00:08:30.160> built<00:08:30.430> Geth<00:08:30.700> it's<00:08:30.880> a<00:08:30.940> great<00:08:31.120> piece 00:08:31.260 --> 00:08:31.270 align:start position:0% guys that built Geth it's a great piece 00:08:31.270 --> 00:08:34.320 align:start position:0% guys that built Geth it's a great piece of<00:08:31.300> software<00:08:31.720> but<00:08:32.500> we<00:08:32.979> just<00:08:33.220> operated<00:08:34.030> it<00:08:34.150> at<00:08:34.240> a 00:08:34.320 --> 00:08:34.330 align:start position:0% of software but we just operated it at a 00:08:34.330 --> 00:08:36.719 align:start position:0% of software but we just operated it at a scale<00:08:34.479> that<00:08:34.660> really<00:08:35.290> no<00:08:35.440> one<00:08:35.590> else<00:08:35.710> had<00:08:36.400> been 00:08:36.719 --> 00:08:36.729 align:start position:0% scale that really no one else had been 00:08:36.729 --> 00:08:39.089 align:start position:0% scale that really no one else had been publicly<00:08:37.000> talking<00:08:37.300> about<00:08:37.620> so<00:08:38.620> the<00:08:38.860> first 00:08:39.089 --> 00:08:39.099 align:start position:0% publicly talking about so the first 00:08:39.099 --> 00:08:40.530 align:start position:0% publicly talking about so the first problem<00:08:39.250> we<00:08:39.400> had<00:08:39.550> was<00:08:39.729> GAF<00:08:39.970> wasn't<00:08:40.390> written 00:08:40.530 --> 00:08:40.540 align:start position:0% problem we had was GAF wasn't written 00:08:40.540 --> 00:08:43.770 align:start position:0% problem we had was GAF wasn't written with<00:08:40.840> modern<00:08:41.229> pub/sub<00:08:41.770> methods<00:08:42.510> this<00:08:43.510> meant 00:08:43.770 --> 00:08:43.780 align:start position:0% with modern pub/sub methods this meant 00:08:43.780 --> 00:08:45.180 align:start position:0% with modern pub/sub methods this meant that<00:08:44.020> you<00:08:44.110> have<00:08:44.260> to<00:08:44.380> do<00:08:44.470> heavy<00:08:44.650> frequent 00:08:45.180 --> 00:08:45.190 align:start position:0% that you have to do heavy frequent 00:08:45.190 --> 00:08:46.980 align:start position:0% that you have to do heavy frequent polling<00:08:45.580> in<00:08:45.730> order<00:08:45.970> to<00:08:46.120> get<00:08:46.270> state<00:08:46.570> changes 00:08:46.980 --> 00:08:46.990 align:start position:0% polling in order to get state changes 00:08:46.990 --> 00:08:49.470 align:start position:0% polling in order to get state changes and<00:08:47.200> you<00:08:47.950> can't<00:08:48.190> just<00:08:48.310> simply<00:08:48.490> subscribe<00:08:49.210> to 00:08:49.470 --> 00:08:49.480 align:start position:0% and you can't just simply subscribe to 00:08:49.480 --> 00:08:51.450 align:start position:0% and you can't just simply subscribe to the<00:08:49.600> events<00:08:50.080> being<00:08:50.590> emitted<00:08:50.890> out<00:08:51.160> of<00:08:51.400> the 00:08:51.450 --> 00:08:51.460 align:start position:0% the events being emitted out of the 00:08:51.460 --> 00:08:53.070 align:start position:0% the events being emitted out of the blockchain<00:08:51.670> that<00:08:51.970> you<00:08:52.090> want<00:08:52.270> to<00:08:52.330> get 00:08:53.070 --> 00:08:53.080 align:start position:0% blockchain that you want to get 00:08:53.080 --> 00:08:55.230 align:start position:0% blockchain that you want to get now<00:08:53.950> that<00:08:54.130> that's<00:08:54.310> kind<00:08:54.550> of<00:08:54.640> true<00:08:54.790> because<00:08:55.060> you 00:08:55.230 --> 00:08:55.240 align:start position:0% now that that's kind of true because you 00:08:55.240 --> 00:08:58.440 align:start position:0% now that that's kind of true because you can<00:08:55.690> with<00:08:55.870> filters<00:08:56.610> but<00:08:57.610> in<00:08:57.820> and<00:08:58.209> I've<00:08:58.269> heard<00:08:58.420> a 00:08:58.440 --> 00:08:58.450 align:start position:0% can with filters but in and I've heard a 00:08:58.450 --> 00:08:59.850 align:start position:0% can with filters but in and I've heard a rumor<00:08:58.690> here<00:08:58.839> that<00:08:58.990> this<00:08:59.260> may<00:08:59.440> no<00:08:59.560> longer<00:08:59.589> be 00:08:59.850 --> 00:08:59.860 align:start position:0% rumor here that this may no longer be 00:08:59.860 --> 00:09:02.310 align:start position:0% rumor here that this may no longer be true<00:09:00.279> but<00:09:00.490> at<00:09:00.610> the<00:09:00.700> time<00:09:01.140> filters<00:09:02.140> were 00:09:02.310 --> 00:09:02.320 align:start position:0% true but at the time filters were 00:09:02.320 --> 00:09:05.940 align:start position:0% true but at the time filters were dropped<00:09:02.709> on<00:09:03.010> a<00:09:03.070> node<00:09:03.399> resetting<00:09:04.029> and<00:09:04.950> you'll 00:09:05.940 --> 00:09:05.950 align:start position:0% dropped on a node resetting and you'll 00:09:05.950 --> 00:09:07.139 align:start position:0% dropped on a node resetting and you'll find<00:09:06.279> out<00:09:06.430> in<00:09:06.519> a<00:09:06.550> couple<00:09:06.580> slides<00:09:06.940> whoa<00:09:07.060> that 00:09:07.139 --> 00:09:07.149 align:start position:0% find out in a couple slides whoa that 00:09:07.149 --> 00:09:09.750 align:start position:0% find out in a couple slides whoa that was<00:09:07.240> a<00:09:07.300> huge<00:09:07.510> issue<00:09:07.720> for<00:09:07.870> us<00:09:08.279> but<00:09:09.279> basically 00:09:09.750 --> 00:09:09.760 align:start position:0% was a huge issue for us but basically 00:09:09.760 --> 00:09:12.329 align:start position:0% was a huge issue for us but basically the<00:09:10.390> filters<00:09:10.779> being<00:09:11.019> dropped<00:09:11.410> was<00:09:11.829> a 00:09:12.329 --> 00:09:12.339 align:start position:0% the filters being dropped was a 00:09:12.339 --> 00:09:15.180 align:start position:0% the filters being dropped was a hindrance<00:09:13.120> to<00:09:13.329> us<00:09:13.589> properly<00:09:14.589> getting<00:09:14.860> a 00:09:15.180 --> 00:09:15.190 align:start position:0% hindrance to us properly getting a 00:09:15.190 --> 00:09:18.329 align:start position:0% hindrance to us properly getting a pub/sub<00:09:15.940> system<00:09:16.540> setup<00:09:16.779> and<00:09:17.200> then<00:09:18.070> further 00:09:18.329 --> 00:09:18.339 align:start position:0% pub/sub system setup and then further 00:09:18.339 --> 00:09:19.560 align:start position:0% pub/sub system setup and then further complicating<00:09:18.820> that<00:09:18.910> is<00:09:19.029> filters<00:09:19.329> aren't<00:09:19.480> a 00:09:19.560 --> 00:09:19.570 align:start position:0% complicating that is filters aren't a 00:09:19.570 --> 00:09:22.290 align:start position:0% complicating that is filters aren't a single<00:09:19.839> node<00:09:19.990> only<00:09:20.370> so<00:09:21.370> when<00:09:21.760> you<00:09:21.880> cost<00:09:22.149> your 00:09:22.290 --> 00:09:22.300 align:start position:0% single node only so when you cost your 00:09:22.300 --> 00:09:24.090 align:start position:0% single node only so when you cost your nose<00:09:22.480> behind<00:09:22.750> a<00:09:22.779> load<00:09:22.959> balancer<00:09:23.110> the 00:09:24.090 --> 00:09:24.100 align:start position:0% nose behind a load balancer the 00:09:24.100 --> 00:09:25.800 align:start position:0% nose behind a load balancer the round-robin<00:09:24.610> aspect<00:09:25.120> the<00:09:25.240> load<00:09:25.420> balancer 00:09:25.800 --> 00:09:25.810 align:start position:0% round-robin aspect the load balancer 00:09:25.810 --> 00:09:27.120 align:start position:0% round-robin aspect the load balancer means<00:09:25.959> you<00:09:26.079> can<00:09:26.230> no<00:09:26.290> longer<00:09:26.470> guarantee<00:09:26.920> which 00:09:27.120 --> 00:09:27.130 align:start position:0% means you can no longer guarantee which 00:09:27.130 --> 00:09:29.639 align:start position:0% means you can no longer guarantee which node<00:09:27.370> is<00:09:27.399> being<00:09:27.700> hit<00:09:28.029> and<00:09:28.269> if<00:09:29.050> the<00:09:29.200> filter<00:09:29.470> even 00:09:29.639 --> 00:09:29.649 align:start position:0% node is being hit and if the filter even 00:09:29.649 --> 00:09:32.370 align:start position:0% node is being hit and if the filter even existed<00:09:30.130> there<00:09:30.690> one<00:09:31.690> of<00:09:31.720> the<00:09:31.839> solutions<00:09:32.260> we 00:09:32.370 --> 00:09:32.380 align:start position:0% existed there one of the solutions we 00:09:32.380 --> 00:09:34.590 align:start position:0% existed there one of the solutions we found<00:09:32.589> at<00:09:32.709> the<00:09:32.769> time<00:09:33.010> was<00:09:33.610> a<00:09:33.640> node.js<00:09:33.910> library 00:09:34.590 --> 00:09:34.600 align:start position:0% found at the time was a node.js library 00:09:34.600 --> 00:09:37.079 align:start position:0% found at the time was a node.js library that<00:09:34.720> basically<00:09:35.050> allowed<00:09:35.320> you<00:09:35.589> to<00:09:35.620> do<00:09:36.010> a<00:09:36.310> multi 00:09:37.079 --> 00:09:37.089 align:start position:0% that basically allowed you to do a multi 00:09:37.089 --> 00:09:39.060 align:start position:0% that basically allowed you to do a multi node<00:09:37.240> filter<00:09:37.630> setup<00:09:37.899> and<00:09:38.110> all<00:09:38.589> it<00:09:38.709> would<00:09:38.829> do<00:09:38.920> is 00:09:39.060 --> 00:09:39.070 align:start position:0% node filter setup and all it would do is 00:09:39.070 --> 00:09:41.250 align:start position:0% node filter setup and all it would do is iterate<00:09:39.220> over<00:09:39.430> an<00:09:39.730> array<00:09:39.820> of<00:09:40.029> guest<00:09:40.779> nodes<00:09:41.019> and 00:09:41.250 --> 00:09:41.260 align:start position:0% iterate over an array of guest nodes and 00:09:41.260 --> 00:09:43.440 align:start position:0% iterate over an array of guest nodes and set<00:09:41.440> up<00:09:41.529> filters<00:09:41.709> on<00:09:41.950> each<00:09:42.130> node<00:09:42.490> it<00:09:43.240> was<00:09:43.420> a 00:09:43.440 --> 00:09:43.450 align:start position:0% set up filters on each node it was a 00:09:43.450 --> 00:09:45.540 align:start position:0% set up filters on each node it was a heavy<00:09:43.750> client-side<00:09:44.320> solution<00:09:44.920> and<00:09:45.100> we<00:09:45.370> really 00:09:45.540 --> 00:09:45.550 align:start position:0% heavy client-side solution and we really 00:09:45.550 --> 00:09:46.319 align:start position:0% heavy client-side solution and we really didn't<00:09:45.730> think<00:09:45.760> that<00:09:45.910> that<00:09:46.029> was<00:09:46.120> what<00:09:46.240> we 00:09:46.319 --> 00:09:46.329 align:start position:0% didn't think that that was what we 00:09:46.329 --> 00:09:48.449 align:start position:0% didn't think that that was what we wanted<00:09:46.510> to<00:09:46.720> do<00:09:47.019> we<00:09:47.709> were<00:09:47.800> hoping<00:09:48.070> for<00:09:48.130> more<00:09:48.399> of 00:09:48.449 --> 00:09:48.459 align:start position:0% wanted to do we were hoping for more of 00:09:48.459 --> 00:09:49.980 align:start position:0% wanted to do we were hoping for more of a<00:09:48.550> server-side<00:09:49.029> solution<00:09:49.570> rather<00:09:49.810> than 00:09:49.980 --> 00:09:49.990 align:start position:0% a server-side solution rather than 00:09:49.990 --> 00:09:53.250 align:start position:0% a server-side solution rather than relying<00:09:50.230> on<00:09:50.410> a<00:09:50.470> client<00:09:50.860> implementation<00:09:52.260> so 00:09:53.250 --> 00:09:53.260 align:start position:0% relying on a client implementation so 00:09:53.260 --> 00:09:54.780 align:start position:0% relying on a client implementation so the<00:09:53.350> underlying<00:09:53.680> issue<00:09:53.950> that<00:09:54.160> we<00:09:54.279> found<00:09:54.640> in 00:09:54.780 --> 00:09:54.790 align:start position:0% the underlying issue that we found in 00:09:54.790 --> 00:09:55.860 align:start position:0% the underlying issue that we found in some<00:09:54.820> of<00:09:54.910> the<00:09:55.000> people<00:09:55.180> have<00:09:55.300> alluded<00:09:55.480> to<00:09:55.750> is 00:09:55.860 --> 00:09:55.870 align:start position:0% some of the people have alluded to is 00:09:55.870 --> 00:09:58.079 align:start position:0% some of the people have alluded to is the<00:09:55.990> overall<00:09:56.440> ecosystem<00:09:56.950> of<00:09:57.430> ethereum<00:09:57.760> is 00:09:58.079 --> 00:09:58.089 align:start position:0% the overall ecosystem of ethereum is 00:09:58.089 --> 00:10:00.180 align:start position:0% the overall ecosystem of ethereum is meant<00:09:58.300> to<00:09:58.420> be<00:09:58.510> fault<00:09:58.810> tolerant<00:09:59.380> when<00:09:59.829> it<00:09:59.980> comes 00:10:00.180 --> 00:10:00.190 align:start position:0% meant to be fault tolerant when it comes 00:10:00.190 --> 00:10:04.319 align:start position:0% meant to be fault tolerant when it comes to<00:10:00.370> chain<00:10:00.579> State<00:10:01.290> however<00:10:02.290> there's<00:10:03.040> no<00:10:03.700> real 00:10:04.319 --> 00:10:04.329 align:start position:0% to chain State however there's no real 00:10:04.329 --> 00:10:05.880 align:start position:0% to chain State however there's no real thought<00:10:04.630> that's<00:10:04.839> been<00:10:05.020> given<00:10:05.260> to<00:10:05.380> the<00:10:05.560> notion 00:10:05.880 --> 00:10:05.890 align:start position:0% thought that's been given to the notion 00:10:05.890 --> 00:10:07.860 align:start position:0% thought that's been given to the notion of<00:10:05.950> creating<00:10:06.130> a<00:10:06.370> subset<00:10:06.910> of<00:10:06.940> nodes<00:10:07.270> that<00:10:07.570> are 00:10:07.860 --> 00:10:07.870 align:start position:0% of creating a subset of nodes that are 00:10:07.870 --> 00:10:09.630 align:start position:0% of creating a subset of nodes that are also<00:10:08.140> fault<00:10:08.709> tolerant<00:10:09.220> in<00:10:09.339> and<00:10:09.490> of<00:10:09.550> themselves 00:10:09.630 --> 00:10:09.640 align:start position:0% also fault tolerant in and of themselves 00:10:09.640 --> 00:10:13.680 align:start position:0% also fault tolerant in and of themselves and<00:10:10.360> so<00:10:11.350> with<00:10:12.040> adapt<00:10:12.399> this<00:10:12.700> means<00:10:13.089> either 00:10:13.680 --> 00:10:13.690 align:start position:0% and so with adapt this means either 00:10:13.690 --> 00:10:16.110 align:start position:0% and so with adapt this means either querying<00:10:14.440> a<00:10:14.620> single<00:10:14.980> node<00:10:15.130> and<00:10:15.399> praying<00:10:15.850> that 00:10:16.110 --> 00:10:16.120 align:start position:0% querying a single node and praying that 00:10:16.120 --> 00:10:18.090 align:start position:0% querying a single node and praying that it<00:10:16.209> doesn't<00:10:16.450> go<00:10:16.660> down<00:10:16.839> using<00:10:17.800> a<00:10:17.860> bunch<00:10:18.040> of 00:10:18.090 --> 00:10:18.100 align:start position:0% it doesn't go down using a bunch of 00:10:18.100 --> 00:10:19.560 align:start position:0% it doesn't go down using a bunch of third<00:10:18.310> party<00:10:18.670> nodes<00:10:18.880> and<00:10:19.089> hoping<00:10:19.329> you<00:10:19.390> have 00:10:19.560 --> 00:10:19.570 align:start position:0% third party nodes and hoping you have 00:10:19.570 --> 00:10:22.560 align:start position:0% third party nodes and hoping you have the<00:10:19.690> right<00:10:19.810> state<00:10:20.170> or<00:10:21.029> doing<00:10:22.029> what<00:10:22.180> in<00:10:22.300> fira 00:10:22.560 --> 00:10:22.570 align:start position:0% the right state or doing what in fira 00:10:22.570 --> 00:10:26.100 align:start position:0% the right state or doing what in fira did<00:10:22.870> which<00:10:23.829> is<00:10:24.160> writing<00:10:25.120> a<00:10:25.329> really<00:10:25.600> cool<00:10:25.870> piece 00:10:26.100 --> 00:10:26.110 align:start position:0% did which is writing a really cool piece 00:10:26.110 --> 00:10:28.829 align:start position:0% did which is writing a really cool piece of<00:10:26.140> software<00:10:26.740> and<00:10:26.860> hardware<00:10:27.300> called<00:10:28.300> ferryman 00:10:28.829 --> 00:10:28.839 align:start position:0% of software and hardware called ferryman 00:10:28.839 --> 00:10:32.370 align:start position:0% of software and hardware called ferryman which<00:10:29.589> basically<00:10:30.130> takes<00:10:31.170> introspects<00:10:32.170> the 00:10:32.370 --> 00:10:32.380 align:start position:0% which basically takes introspects the 00:10:32.380 --> 00:10:35.069 align:start position:0% which basically takes introspects the request<00:10:32.770> coming<00:10:33.040> in<00:10:33.310> and<00:10:33.550> then<00:10:34.209> it<00:10:34.540> sends<00:10:34.990> the 00:10:35.069 --> 00:10:35.079 align:start position:0% request coming in and then it sends the 00:10:35.079 --> 00:10:37.280 align:start position:0% request coming in and then it sends the request<00:10:35.230> off<00:10:35.649> to<00:10:35.829> the<00:10:35.920> proper<00:10:36.250> type<00:10:36.430> of<00:10:36.610> node 00:10:37.280 --> 00:10:37.290 align:start position:0% request off to the proper type of node 00:10:37.290 --> 00:10:39.960 align:start position:0% request off to the proper type of node also<00:10:38.290> provides<00:10:38.589> a<00:10:38.740> modern<00:10:38.920> pub<00:10:39.250> sub<00:10:39.550> layer<00:10:39.730> on 00:10:39.960 --> 00:10:39.970 align:start position:0% also provides a modern pub sub layer on 00:10:39.970 --> 00:10:43.380 align:start position:0% also provides a modern pub sub layer on top<00:10:40.209> of<00:10:40.270> that<00:10:40.740> and<00:10:41.740> then<00:10:41.860> allows<00:10:42.160> for<00:10:42.850> you<00:10:43.209> to 00:10:43.380 --> 00:10:43.390 align:start position:0% top of that and then allows for you to 00:10:43.390 --> 00:10:45.960 align:start position:0% top of that and then allows for you to subscribe<00:10:43.600> to<00:10:44.050> client<00:10:44.380> events<00:10:44.740> the<00:10:45.520> problem 00:10:45.960 --> 00:10:45.970 align:start position:0% subscribe to client events the problem 00:10:45.970 --> 00:10:47.730 align:start position:0% subscribe to client events the problem is<00:10:46.089> the<00:10:46.300> inferior<00:10:46.600> solution<00:10:47.110> is<00:10:47.470> very 00:10:47.730 --> 00:10:47.740 align:start position:0% is the inferior solution is very 00:10:47.740 --> 00:10:50.189 align:start position:0% is the inferior solution is very proprietary<00:10:48.520> and<00:10:48.820> we<00:10:49.149> make<00:10:49.329> sense<00:10:49.630> for<00:10:49.899> their 00:10:50.189 --> 00:10:50.199 align:start position:0% proprietary and we make sense for their 00:10:50.199 --> 00:10:52.380 align:start position:0% proprietary and we make sense for their setup<00:10:50.680> so<00:10:51.310> they<00:10:51.430> can't<00:10:51.670> really<00:10:51.790> open<00:10:52.180> source 00:10:52.380 --> 00:10:52.390 align:start position:0% setup so they can't really open source 00:10:52.390 --> 00:10:56.910 align:start position:0% setup so they can't really open source that<00:10:52.630> to<00:10:52.779> the<00:10:52.839> community<00:10:54.990> our<00:10:55.990> story<00:10:56.620> on<00:10:56.890> our 00:10:56.910 --> 00:10:56.920 align:start position:0% that to the community our story on our 00:10:56.920 --> 00:11:00.600 align:start position:0% that to the community our story on our gaff<00:10:57.399> nodes<00:10:58.199> we<00:10:59.199> do<00:10:59.410> about<00:10:59.680> 800<00:11:00.130> requests<00:11:00.430> per 00:11:00.600 --> 00:11:00.610 align:start position:0% gaff nodes we do about 800 requests per 00:11:00.610 --> 00:11:04.260 align:start position:0% gaff nodes we do about 800 requests per second<00:11:01.000> on<00:11:01.180> our<00:11:01.500> gatherings<00:11:03.029> requests<00:11:04.029> per 00:11:04.260 --> 00:11:04.270 align:start position:0% second on our gatherings requests per 00:11:04.270 --> 00:11:06.220 align:start position:0% second on our gatherings requests per day<00:11:04.470> that's<00:11:05.470> over 00:11:06.220 --> 00:11:06.230 align:start position:0% day that's over 00:11:06.230 --> 00:11:08.470 align:start position:0% day that's over to<00:11:06.350> clusters<00:11:06.800> of<00:11:06.830> three<00:11:07.040> nodes<00:11:07.280> each<00:11:07.480> so 00:11:08.470 --> 00:11:08.480 align:start position:0% to clusters of three nodes each so 00:11:08.480 --> 00:11:10.120 align:start position:0% to clusters of three nodes each so that's<00:11:08.780> a<00:11:08.990> fairly<00:11:09.260> large<00:11:09.470> amount<00:11:09.650> of<00:11:09.920> traffic 00:11:10.120 --> 00:11:10.130 align:start position:0% that's a fairly large amount of traffic 00:11:10.130 --> 00:11:14.700 align:start position:0% that's a fairly large amount of traffic and<00:11:11.020> under<00:11:12.020> this<00:11:12.140> load<00:11:12.500> death<00:11:13.250> actually<00:11:14.030> broke 00:11:14.700 --> 00:11:14.710 align:start position:0% and under this load death actually broke 00:11:14.710 --> 00:11:18.310 align:start position:0% and under this load death actually broke we<00:11:15.710> had<00:11:16.130> a<00:11:16.160> massive<00:11:16.610> memory<00:11:17.210> usage<00:11:17.630> issue<00:11:17.900> and 00:11:18.310 --> 00:11:18.320 align:start position:0% we had a massive memory usage issue and 00:11:18.320 --> 00:11:20.530 align:start position:0% we had a massive memory usage issue and a<00:11:18.350> recurring<00:11:18.800> memory<00:11:19.100> leak<00:11:19.340> that<00:11:19.880> caused<00:11:20.210> each 00:11:20.530 --> 00:11:20.540 align:start position:0% a recurring memory leak that caused each 00:11:20.540 --> 00:11:22.600 align:start position:0% a recurring memory leak that caused each node<00:11:20.810> to<00:11:20.990> slowly<00:11:21.260> utilize<00:11:21.770> all<00:11:22.160> the<00:11:22.460> RAM 00:11:22.600 --> 00:11:22.610 align:start position:0% node to slowly utilize all the RAM 00:11:22.610 --> 00:11:24.310 align:start position:0% node to slowly utilize all the RAM available<00:11:23.330> on<00:11:23.480> the<00:11:23.720> box 00:11:24.310 --> 00:11:24.320 align:start position:0% available on the box 00:11:24.320 --> 00:11:27.880 align:start position:0% available on the box we<00:11:25.280> tried<00:11:25.850> scaling<00:11:26.210> up<00:11:26.330> our<00:11:26.480> hardware<00:11:26.840> -<00:11:27.170> this 00:11:27.880 --> 00:11:27.890 align:start position:0% we tried scaling up our hardware - this 00:11:27.890 --> 00:11:30.820 align:start position:0% we tried scaling up our hardware - this is<00:11:28.010> AWS<00:11:28.490> terms<00:11:28.730> but<00:11:28.940> m-52<00:11:29.540> X<00:11:29.720> larges<00:11:30.140> so<00:11:30.650> it's<00:11:30.770> a 00:11:30.820 --> 00:11:30.830 align:start position:0% is AWS terms but m-52 X larges so it's a 00:11:30.830 --> 00:11:33.880 align:start position:0% is AWS terms but m-52 X larges so it's a 32<00:11:31.220> gig<00:11:31.490> of<00:11:31.610> ram<00:11:31.640> 8<00:11:31.910> v<00:11:32.150> cpu<00:11:32.780> piece<00:11:33.440> of<00:11:33.590> hardware 00:11:33.880 --> 00:11:33.890 align:start position:0% 32 gig of ram 8 v cpu piece of hardware 00:11:33.890 --> 00:11:37.270 align:start position:0% 32 gig of ram 8 v cpu piece of hardware and<00:11:34.390> with<00:11:35.390> six<00:11:35.720> of<00:11:35.840> those<00:11:35.960> in<00:11:36.200> production<00:11:36.800> you 00:11:37.270 --> 00:11:37.280 align:start position:0% and with six of those in production you 00:11:37.280 --> 00:11:39.580 align:start position:0% and with six of those in production you should<00:11:37.610> be<00:11:37.760> able<00:11:37.940> to<00:11:38.090> do<00:11:38.360> substantially<00:11:39.320> more 00:11:39.580 --> 00:11:39.590 align:start position:0% should be able to do substantially more 00:11:39.590 --> 00:11:42.610 align:start position:0% should be able to do substantially more than<00:11:39.920> 70<00:11:40.610> million<00:11:40.790> requests<00:11:41.240> per<00:11:41.270> day<00:11:41.530> but<00:11:42.530> as 00:11:42.610 --> 00:11:42.620 align:start position:0% than 70 million requests per day but as 00:11:42.620 --> 00:11:44.680 align:start position:0% than 70 million requests per day but as you<00:11:42.710> can<00:11:42.800> see<00:11:42.860> from<00:11:43.100> the<00:11:43.160> chart<00:11:43.190> here<00:11:43.730> after 00:11:44.680 --> 00:11:44.690 align:start position:0% you can see from the chart here after 00:11:44.690 --> 00:11:46.300 align:start position:0% you can see from the chart here after about<00:11:44.840> 15<00:11:45.260> hours<00:11:45.410> we<00:11:45.680> had<00:11:45.770> topped<00:11:46.040> out<00:11:46.190> our 00:11:46.300 --> 00:11:46.310 align:start position:0% about 15 hours we had topped out our 00:11:46.310 --> 00:11:49.120 align:start position:0% about 15 hours we had topped out our memory<00:11:46.580> right<00:11:47.210> at<00:11:47.330> 32<00:11:47.720> gigs<00:11:48.020> and<00:11:48.290> the<00:11:48.860> drops 00:11:49.120 --> 00:11:49.130 align:start position:0% memory right at 32 gigs and the drops 00:11:49.130 --> 00:11:50.440 align:start position:0% memory right at 32 gigs and the drops you<00:11:49.340> see<00:11:49.520> where<00:11:49.730> we<00:11:49.820> actually<00:11:50.210> went<00:11:50.360> in 00:11:50.440 --> 00:11:50.450 align:start position:0% you see where we actually went in 00:11:50.450 --> 00:11:51.760 align:start position:0% you see where we actually went in physically<00:11:50.960> and<00:11:51.080> restarted<00:11:51.560> the<00:11:51.620> goth 00:11:51.760 --> 00:11:51.770 align:start position:0% physically and restarted the goth 00:11:51.770 --> 00:11:57.310 align:start position:0% physically and restarted the goth process<00:11:54.220> the<00:11:55.220> memory<00:11:56.150> leak<00:11:56.300> meant<00:11:56.690> that<00:11:57.170> we 00:11:57.310 --> 00:11:57.320 align:start position:0% process the memory leak meant that we 00:11:57.320 --> 00:11:58.870 align:start position:0% process the memory leak meant that we had<00:11:57.440> to<00:11:57.470> cycle<00:11:57.890> our<00:11:57.980> gas<00:11:58.130> nodes<00:11:58.370> before<00:11:58.610> the 00:11:58.870 --> 00:11:58.880 align:start position:0% had to cycle our gas nodes before the 00:11:58.880 --> 00:12:01.030 align:start position:0% had to cycle our gas nodes before the GAF<00:11:59.030> processes<00:11:59.660> used<00:12:00.050> all<00:12:00.320> the<00:12:00.470> available<00:12:00.680> ram 00:12:01.030 --> 00:12:01.040 align:start position:0% GAF processes used all the available ram 00:12:01.040 --> 00:12:03.760 align:start position:0% GAF processes used all the available ram in<00:12:01.880> fact<00:12:02.150> if<00:12:02.270> we<00:12:02.390> didn't<00:12:02.690> catch<00:12:02.900> it<00:12:03.080> in<00:12:03.170> time 00:12:03.760 --> 00:12:03.770 align:start position:0% in fact if we didn't catch it in time 00:12:03.770 --> 00:12:07.510 align:start position:0% in fact if we didn't catch it in time the<00:12:04.190> box<00:12:04.460> itself<00:12:04.700> was<00:12:05.540> so<00:12:06.100> the<00:12:07.100> ram<00:12:07.220> was<00:12:07.370> still 00:12:07.510 --> 00:12:07.520 align:start position:0% the box itself was so the ram was still 00:12:07.520 --> 00:12:09.340 align:start position:0% the box itself was so the ram was still used<00:12:07.760> up<00:12:07.880> you<00:12:08.030> couldn't<00:12:08.300> even<00:12:08.390> SSH<00:12:08.930> into<00:12:09.260> the 00:12:09.340 --> 00:12:09.350 align:start position:0% used up you couldn't even SSH into the 00:12:09.350 --> 00:12:11.800 align:start position:0% used up you couldn't even SSH into the box<00:12:09.590> to<00:12:10.280> kill<00:12:10.460> the<00:12:10.580> GAF<00:12:10.700> process<00:12:11.180> you<00:12:11.600> had<00:12:11.720> to 00:12:11.800 --> 00:12:11.810 align:start position:0% box to kill the GAF process you had to 00:12:11.810 --> 00:12:15.670 align:start position:0% box to kill the GAF process you had to restart<00:12:12.200> the<00:12:12.500> whole<00:12:12.710> node<00:12:13.520> itself<00:12:13.880> and<00:12:14.680> this 00:12:15.670 --> 00:12:15.680 align:start position:0% restart the whole node itself and this 00:12:15.680 --> 00:12:17.080 align:start position:0% restart the whole node itself and this meant<00:12:15.920> that<00:12:16.040> we<00:12:16.130> were<00:12:16.190> waking<00:12:16.370> up<00:12:16.670> every<00:12:16.970> three 00:12:17.080 --> 00:12:17.090 align:start position:0% meant that we were waking up every three 00:12:17.090 --> 00:12:18.490 align:start position:0% meant that we were waking up every three to<00:12:17.240> four<00:12:17.390> hours<00:12:17.450> to<00:12:18.020> check<00:12:18.170> memory 00:12:18.490 --> 00:12:18.500 align:start position:0% to four hours to check memory 00:12:18.500 --> 00:12:20.500 align:start position:0% to four hours to check memory consumption<00:12:18.920> and<00:12:19.310> manually<00:12:20.090> we<00:12:20.210> start<00:12:20.420> the 00:12:20.500 --> 00:12:20.510 align:start position:0% consumption and manually we start the 00:12:20.510 --> 00:12:21.730 align:start position:0% consumption and manually we start the process<00:12:20.720> to<00:12:21.110> make<00:12:21.230> sure<00:12:21.380> we<00:12:21.470> didn't<00:12:21.650> get 00:12:21.730 --> 00:12:21.740 align:start position:0% process to make sure we didn't get 00:12:21.740 --> 00:12:24.580 align:start position:0% process to make sure we didn't get locked<00:12:21.950> out<00:12:22.100> of<00:12:22.130> our<00:12:22.220> boxes<00:12:22.730> and<00:12:23.500> once<00:12:24.500> we 00:12:24.580 --> 00:12:24.590 align:start position:0% locked out of our boxes and once we 00:12:24.590 --> 00:12:26.380 align:start position:0% locked out of our boxes and once we figure<00:12:24.710> that's<00:12:24.950> the<00:12:24.980> usual<00:12:25.190> cycle<00:12:25.820> of<00:12:26.000> RAM 00:12:26.380 --> 00:12:26.390 align:start position:0% figure that's the usual cycle of RAM 00:12:26.390 --> 00:12:28.600 align:start position:0% figure that's the usual cycle of RAM usage<00:12:26.630> we<00:12:27.050> were<00:12:27.140> able<00:12:27.320> to<00:12:27.380> write<00:12:27.560> cron<00:12:27.830> jobs<00:12:28.160> to 00:12:28.600 --> 00:12:28.610 align:start position:0% usage we were able to write cron jobs to 00:12:28.610 --> 00:12:31.690 align:start position:0% usage we were able to write cron jobs to then<00:12:28.730> go<00:12:28.910> in<00:12:29.120> and<00:12:29.420> restart<00:12:30.320> our<00:12:30.500> Geth<00:12:30.710> process 00:12:31.690 --> 00:12:31.700 align:start position:0% then go in and restart our Geth process 00:12:31.700 --> 00:12:33.790 align:start position:0% then go in and restart our Geth process and<00:12:32.030> we<00:12:32.480> staged<00:12:32.750> that<00:12:32.990> over<00:12:33.290> about<00:12:33.350> an<00:12:33.590> hour 00:12:33.790 --> 00:12:33.800 align:start position:0% and we staged that over about an hour 00:12:33.800 --> 00:12:37.570 align:start position:0% and we staged that over about an hour across<00:12:34.220> six<00:12:34.610> nodes<00:12:34.880> and<00:12:35.470> it<00:12:36.470> worked<00:12:36.740> but<00:12:37.400> for 00:12:37.570 --> 00:12:37.580 align:start position:0% across six nodes and it worked but for 00:12:37.580 --> 00:12:38.350 align:start position:0% across six nodes and it worked but for any<00:12:37.700> of<00:12:37.760> you<00:12:37.910> guys<00:12:38.000> that<00:12:38.150> work<00:12:38.300> in 00:12:38.350 --> 00:12:38.360 align:start position:0% any of you guys that work in 00:12:38.360 --> 00:12:40.180 align:start position:0% any of you guys that work in infrastructure<00:12:38.870> trusting<00:12:39.860> your<00:12:39.920> entire 00:12:40.180 --> 00:12:40.190 align:start position:0% infrastructure trusting your entire 00:12:40.190 --> 00:12:43.000 align:start position:0% infrastructure trusting your entire system<00:12:40.640> to<00:12:40.670> six<00:12:40.970> cron<00:12:41.240> jobs<00:12:41.570> is<00:12:42.020> incredibly 00:12:43.000 --> 00:12:43.010 align:start position:0% system to six cron jobs is incredibly 00:12:43.010 --> 00:12:46.150 align:start position:0% system to six cron jobs is incredibly scary<00:12:43.480> so<00:12:44.480> we<00:12:45.230> started<00:12:45.620> tinkering<00:12:45.770> with<00:12:45.950> some 00:12:46.150 --> 00:12:46.160 align:start position:0% scary so we started tinkering with some 00:12:46.160 --> 00:12:48.370 align:start position:0% scary so we started tinkering with some of<00:12:46.280> our<00:12:46.400> guest<00:12:47.030> settings<00:12:47.390> and<00:12:47.660> we<00:12:47.870> actually 00:12:48.370 --> 00:12:48.380 align:start position:0% of our guest settings and we actually 00:12:48.380 --> 00:12:50.650 align:start position:0% of our guest settings and we actually dropped<00:12:48.560> our<00:12:48.770> max<00:12:48.950> peers<00:12:49.370> setting<00:12:49.790> and<00:12:49.970> the 00:12:50.650 --> 00:12:50.660 align:start position:0% dropped our max peers setting and the 00:12:50.660 --> 00:12:52.890 align:start position:0% dropped our max peers setting and the memory<00:12:50.930> usage<00:12:51.080> issue<00:12:51.440> was<00:12:51.710> largely<00:12:52.070> resolved 00:12:52.890 --> 00:12:52.900 align:start position:0% memory usage issue was largely resolved 00:12:52.900 --> 00:12:55.270 align:start position:0% memory usage issue was largely resolved and<00:12:53.900> it<00:12:54.020> turned<00:12:54.200> out<00:12:54.290> -<00:12:54.410> the<00:12:54.500> memory<00:12:54.770> leak<00:12:54.920> with 00:12:55.270 --> 00:12:55.280 align:start position:0% and it turned out - the memory leak with 00:12:55.280 --> 00:12:57.070 align:start position:0% and it turned out - the memory leak with the<00:12:55.490> max<00:12:55.700> peer<00:12:56.000> setting<00:12:56.360> really<00:12:56.900> only 00:12:57.070 --> 00:12:57.080 align:start position:0% the max peer setting really only 00:12:57.080 --> 00:12:58.360 align:start position:0% the max peer setting really only occurred<00:12:57.440> at<00:12:57.560> the<00:12:57.650> scale<00:12:57.890> at<00:12:58.010> which<00:12:58.040> we<00:12:58.250> were 00:12:58.360 --> 00:12:58.370 align:start position:0% occurred at the scale at which we were 00:12:58.370 --> 00:13:00.040 align:start position:0% occurred at the scale at which we were operating<00:12:58.730> and<00:12:59.150> it<00:12:59.450> wasn't<00:12:59.630> publicly 00:13:00.040 --> 00:13:00.050 align:start position:0% operating and it wasn't publicly 00:13:00.050 --> 00:13:02.920 align:start position:0% operating and it wasn't publicly documented<00:13:00.380> prior<00:13:00.860> to<00:13:01.040> that<00:13:01.600> as<00:13:02.600> you<00:13:02.660> can<00:13:02.810> see 00:13:02.920 --> 00:13:02.930 align:start position:0% documented prior to that as you can see 00:13:02.930 --> 00:13:04.600 align:start position:0% documented prior to that as you can see in<00:13:02.990> the<00:13:03.080> charts<00:13:03.290> here<00:13:03.530> those<00:13:04.040> those<00:13:04.310> drops 00:13:04.600 --> 00:13:04.610 align:start position:0% in the charts here those those drops 00:13:04.610 --> 00:13:05.980 align:start position:0% in the charts here those those drops like<00:13:04.790> that's<00:13:05.060> when<00:13:05.270> we<00:13:05.420> were<00:13:05.540> restarting 00:13:05.980 --> 00:13:05.990 align:start position:0% like that's when we were restarting 00:13:05.990 --> 00:13:10.330 align:start position:0% like that's when we were restarting death<00:13:08.200> the<00:13:09.200> stability<00:13:09.620> that<00:13:09.710> we<00:13:09.830> did<00:13:10.010> finally 00:13:10.330 --> 00:13:10.340 align:start position:0% death the stability that we did finally 00:13:10.340 --> 00:13:11.830 align:start position:0% death the stability that we did finally get<00:13:10.490> came<00:13:10.760> at<00:13:10.910> an<00:13:11.030> incredible<00:13:11.390> cost<00:13:11.600> to<00:13:11.780> our 00:13:11.830 --> 00:13:11.840 align:start position:0% get came at an incredible cost to our 00:13:11.840 --> 00:13:14.950 align:start position:0% get came at an incredible cost to our operations<00:13:12.350> team<00:13:12.590> so<00:13:13.490> for<00:13:14.270> at<00:13:14.540> least<00:13:14.630> two<00:13:14.840> of 00:13:14.950 --> 00:13:14.960 align:start position:0% operations team so for at least two of 00:13:14.960 --> 00:13:16.090 align:start position:0% operations team so for at least two of us<00:13:15.050> we<00:13:15.200> were<00:13:15.260> getting<00:13:15.440> nothing<00:13:15.710> done<00:13:15.740> during 00:13:16.090 --> 00:13:16.100 align:start position:0% us we were getting nothing done during 00:13:16.100 --> 00:13:19.450 align:start position:0% us we were getting nothing done during the<00:13:16.280> day<00:13:16.310> other<00:13:16.850> than<00:13:17.450> goth<00:13:17.600> node<00:13:17.840> work<00:13:18.110> and<00:13:18.460> I 00:13:19.450 --> 00:13:19.460 align:start position:0% the day other than goth node work and I 00:13:19.460 --> 00:13:19.840 align:start position:0% the day other than goth node work and I can't 00:13:19.840 --> 00:13:19.850 align:start position:0% can't 00:13:19.850 --> 00:13:21.370 align:start position:0% can't that<00:13:20.149> your<00:13:20.269> sleep<00:13:20.449> quality<00:13:20.720> certainly 00:13:21.370 --> 00:13:21.380 align:start position:0% that your sleep quality certainly 00:13:21.380 --> 00:13:23.110 align:start position:0% that your sleep quality certainly suffers<00:13:21.829> when<00:13:22.069> you're<00:13:22.190> waking<00:13:22.310> up<00:13:22.610> every<00:13:22.880> few 00:13:23.110 --> 00:13:23.120 align:start position:0% suffers when you're waking up every few 00:13:23.120 --> 00:13:25.090 align:start position:0% suffers when you're waking up every few hours<00:13:23.480> to<00:13:23.990> make<00:13:24.170> sure<00:13:24.410> that<00:13:24.589> your<00:13:24.740> boxes 00:13:25.090 --> 00:13:25.100 align:start position:0% hours to make sure that your boxes 00:13:25.100 --> 00:13:27.120 align:start position:0% hours to make sure that your boxes aren't<00:13:25.250> freezing<00:13:25.759> or<00:13:26.060> running<00:13:26.569> out<00:13:26.660> of<00:13:26.690> memory 00:13:27.120 --> 00:13:27.130 align:start position:0% aren't freezing or running out of memory 00:13:27.130 --> 00:13:29.800 align:start position:0% aren't freezing or running out of memory also<00:13:28.130> your<00:13:28.579> morale<00:13:28.850> really<00:13:29.209> suffers<00:13:29.600> when 00:13:29.800 --> 00:13:29.810 align:start position:0% also your morale really suffers when 00:13:29.810 --> 00:13:31.629 align:start position:0% also your morale really suffers when every<00:13:30.170> morning<00:13:30.529> before<00:13:30.649> breakfast<00:13:31.279> you<00:13:31.459> wake 00:13:31.629 --> 00:13:31.639 align:start position:0% every morning before breakfast you wake 00:13:31.639 --> 00:13:34.389 align:start position:0% every morning before breakfast you wake up<00:13:31.850> to<00:13:32.089> 50-plus<00:13:32.660> pager<00:13:33.079> duty<00:13:33.440> alerts<00:13:33.680> and<00:13:34.009> they 00:13:34.389 --> 00:13:34.399 align:start position:0% up to 50-plus pager duty alerts and they 00:13:34.399 --> 00:13:36.780 align:start position:0% up to 50-plus pager duty alerts and they just<00:13:34.579> keep<00:13:34.790> rolling<00:13:35.240> in<00:13:35.389> throughout<00:13:35.660> the<00:13:35.720> day 00:13:36.780 --> 00:13:36.790 align:start position:0% just keep rolling in throughout the day 00:13:36.790 --> 00:13:39.879 align:start position:0% just keep rolling in throughout the day so<00:13:37.790> we<00:13:38.269> wanted<00:13:38.540> to<00:13:38.630> fix<00:13:38.810> this<00:13:39.050> and<00:13:39.350> we<00:13:39.500> embarked 00:13:39.879 --> 00:13:39.889 align:start position:0% so we wanted to fix this and we embarked 00:13:39.889 --> 00:13:42.280 align:start position:0% so we wanted to fix this and we embarked any<00:13:40.069> massive<00:13:40.430> experimentation<00:13:41.060> to<00:13:41.899> find<00:13:42.170> the 00:13:42.280 --> 00:13:42.290 align:start position:0% any massive experimentation to find the 00:13:42.290 --> 00:13:43.329 align:start position:0% any massive experimentation to find the correct<00:13:42.589> hardware<00:13:42.769> and<00:13:43.160> software 00:13:43.329 --> 00:13:43.339 align:start position:0% correct hardware and software 00:13:43.339 --> 00:13:44.889 align:start position:0% correct hardware and software combination<00:13:44.240> that<00:13:44.420> would<00:13:44.509> be<00:13:44.630> the<00:13:44.750> most 00:13:44.889 --> 00:13:44.899 align:start position:0% combination that would be the most 00:13:44.899 --> 00:13:46.809 align:start position:0% combination that would be the most stable<00:13:45.230> and<00:13:45.610> cost-effective 00:13:46.809 --> 00:13:46.819 align:start position:0% stable and cost-effective 00:13:46.819 --> 00:13:49.300 align:start position:0% stable and cost-effective we<00:13:47.750> ended<00:13:47.990> up<00:13:48.079> iterating<00:13:48.709> over<00:13:48.769> ten<00:13:49.040> different 00:13:49.300 --> 00:13:49.310 align:start position:0% we ended up iterating over ten different 00:13:49.310 --> 00:13:51.639 align:start position:0% we ended up iterating over ten different instance<00:13:49.940> types<00:13:50.209> spinning<00:13:51.199> up<00:13:51.290> everything 00:13:51.639 --> 00:13:51.649 align:start position:0% instance types spinning up everything 00:13:51.649 --> 00:13:55.030 align:start position:0% instance types spinning up everything from<00:13:51.769> t2<00:13:52.040> micros<00:13:52.730> to<00:13:53.180> four<00:13:53.420> Excel<00:13:53.750> boxes<00:13:54.319> and 00:13:55.030 --> 00:13:55.040 align:start position:0% from t2 micros to four Excel boxes and 00:13:55.040 --> 00:13:57.160 align:start position:0% from t2 micros to four Excel boxes and we<00:13:55.279> tried<00:13:55.459> everything<00:13:55.639> from<00:13:55.839> compute<00:13:56.839> focus 00:13:57.160 --> 00:13:57.170 align:start position:0% we tried everything from compute focus 00:13:57.170 --> 00:13:59.379 align:start position:0% we tried everything from compute focus memory<00:13:57.500> focus<00:13:57.949> instant<00:13:58.579> store<00:13:58.910> focused 00:13:59.379 --> 00:13:59.389 align:start position:0% memory focus instant store focused 00:13:59.389 --> 00:14:02.139 align:start position:0% memory focus instant store focused trying<00:14:00.170> to<00:14:00.290> find<00:14:00.589> and<00:14:00.920> profile<00:14:01.459> the<00:14:01.699> way<00:14:01.850> that 00:14:02.139 --> 00:14:02.149 align:start position:0% trying to find and profile the way that 00:14:02.149 --> 00:14:04.240 align:start position:0% trying to find and profile the way that the<00:14:02.360> Geth<00:14:02.660> process<00:14:03.199> worked<00:14:03.529> under<00:14:03.920> the<00:14:04.040> loads 00:14:04.240 --> 00:14:04.250 align:start position:0% the Geth process worked under the loads 00:14:04.250 --> 00:14:06.660 align:start position:0% the Geth process worked under the loads that<00:14:04.459> we<00:14:04.579> were<00:14:04.699> seeing 00:14:06.660 --> 00:14:06.670 align:start position:0% that we were seeing 00:14:06.670 --> 00:14:08.949 align:start position:0% that we were seeing we<00:14:07.670> made<00:14:07.850> some<00:14:08.000> concessions<00:14:08.180> along<00:14:08.720> the<00:14:08.839> way 00:14:08.949 --> 00:14:08.959 align:start position:0% we made some concessions along the way 00:14:08.959 --> 00:14:11.740 align:start position:0% we made some concessions along the way that<00:14:09.769> we<00:14:10.250> have<00:14:10.670> built<00:14:11.120> into<00:14:11.300> our<00:14:11.449> open<00:14:11.630> source 00:14:11.740 --> 00:14:11.750 align:start position:0% that we have built into our open source 00:14:11.750 --> 00:14:14.949 align:start position:0% that we have built into our open source solution<00:14:12.319> unfortunately<00:14:13.180> no<00:14:14.180> pub/sub<00:14:14.630> method 00:14:14.949 --> 00:14:14.959 align:start position:0% solution unfortunately no pub/sub method 00:14:14.959 --> 00:14:15.879 align:start position:0% solution unfortunately no pub/sub method meant<00:14:15.230> that<00:14:15.350> we<00:14:15.440> were<00:14:15.500> just<00:14:15.589> gonna<00:14:15.709> brute 00:14:15.879 --> 00:14:15.889 align:start position:0% meant that we were just gonna brute 00:14:15.889 --> 00:14:17.980 align:start position:0% meant that we were just gonna brute force<00:14:16.130> it<00:14:16.310> and<00:14:16.490> that<00:14:17.360> was<00:14:17.509> one<00:14:17.660> of<00:14:17.690> our<00:14:17.779> big 00:14:17.980 --> 00:14:17.990 align:start position:0% force it and that was one of our big 00:14:17.990 --> 00:14:20.610 align:start position:0% force it and that was one of our big decisions<00:14:18.019> out<00:14:18.800> of<00:14:19.190> the<00:14:19.339> bat<00:14:19.459> out<00:14:20.180> of<00:14:20.209> the<00:14:20.300> gate 00:14:20.610 --> 00:14:20.620 align:start position:0% decisions out of the bat out of the gate 00:14:20.620 --> 00:14:23.019 align:start position:0% decisions out of the bat out of the gate we<00:14:21.620> couldn't<00:14:21.920> guarantee<00:14:22.040> that<00:14:22.370> a<00:14:22.610> single<00:14:22.910> node 00:14:23.019 --> 00:14:23.029 align:start position:0% we couldn't guarantee that a single node 00:14:23.029 --> 00:14:24.910 align:start position:0% we couldn't guarantee that a single node was<00:14:23.180> up<00:14:23.329> so<00:14:23.839> we<00:14:24.019> needed<00:14:24.170> to<00:14:24.259> load<00:14:24.470> balancer 00:14:24.910 --> 00:14:24.920 align:start position:0% was up so we needed to load balancer 00:14:24.920 --> 00:14:27.220 align:start position:0% was up so we needed to load balancer nodes<00:14:25.389> we<00:14:26.389> couldn't<00:14:26.750> guarantee<00:14:26.870> there<00:14:27.170> are 00:14:27.220 --> 00:14:27.230 align:start position:0% nodes we couldn't guarantee there are 00:14:27.230 --> 00:14:29.650 align:start position:0% nodes we couldn't guarantee there are nodes<00:14:27.350> were<00:14:27.529> in<00:14:27.680> sync<00:14:28.040> and<00:14:28.250> up<00:14:28.639> so<00:14:29.389> we<00:14:29.480> needed 00:14:29.650 --> 00:14:29.660 align:start position:0% nodes were in sync and up so we needed 00:14:29.660 --> 00:14:32.019 align:start position:0% nodes were in sync and up so we needed to<00:14:29.779> write<00:14:29.930> health<00:14:30.170> checks<00:14:30.670> we<00:14:31.670> also<00:14:31.790> couldn't 00:14:32.019 --> 00:14:32.029 align:start position:0% to write health checks we also couldn't 00:14:32.029 --> 00:14:33.639 align:start position:0% to write health checks we also couldn't guarantee<00:14:32.209> that<00:14:32.750> a<00:14:32.779> node<00:14:33.110> was<00:14:33.319> at<00:14:33.410> the<00:14:33.500> latest 00:14:33.639 --> 00:14:33.649 align:start position:0% guarantee that a node was at the latest 00:14:33.649 --> 00:14:36.040 align:start position:0% guarantee that a node was at the latest head<00:14:33.980> so<00:14:34.250> we<00:14:34.339> wanted<00:14:34.610> to<00:14:34.699> do<00:14:34.819> pure<00:14:35.089> checking<00:14:35.540> to 00:14:36.040 --> 00:14:36.050 align:start position:0% head so we wanted to do pure checking to 00:14:36.050 --> 00:14:37.749 align:start position:0% head so we wanted to do pure checking to make<00:14:36.079> sure<00:14:36.259> that<00:14:36.589> we<00:14:36.709> were<00:14:36.800> actually<00:14:36.980> at<00:14:37.490> the 00:14:37.749 --> 00:14:37.759 align:start position:0% make sure that we were actually at the 00:14:37.759 --> 00:14:39.040 align:start position:0% make sure that we were actually at the the<00:14:38.029> head<00:14:38.180> state<00:14:38.420> that<00:14:38.540> we<00:14:38.630> thought<00:14:38.810> we<00:14:38.930> were 00:14:39.040 --> 00:14:39.050 align:start position:0% the head state that we thought we were 00:14:39.050 --> 00:14:40.960 align:start position:0% the head state that we thought we were so<00:14:39.980> that<00:14:40.100> combination<00:14:40.339> was<00:14:40.639> kind<00:14:40.850> of<00:14:40.880> the 00:14:40.960 --> 00:14:40.970 align:start position:0% so that combination was kind of the 00:14:40.970 --> 00:14:42.160 align:start position:0% so that combination was kind of the secret<00:14:41.240> sauce<00:14:41.420> that<00:14:41.630> no<00:14:41.720> one<00:14:41.870> was<00:14:41.959> really 00:14:42.160 --> 00:14:42.170 align:start position:0% secret sauce that no one was really 00:14:42.170 --> 00:14:43.660 align:start position:0% secret sauce that no one was really talking<00:14:42.470> about<00:14:42.500> online<00:14:42.740> which<00:14:43.490> was 00:14:43.660 --> 00:14:43.670 align:start position:0% talking about online which was 00:14:43.670 --> 00:14:45.340 align:start position:0% talking about online which was aggressive<00:14:44.120> health<00:14:44.269> checking<00:14:44.689> aggressive 00:14:45.340 --> 00:14:45.350 align:start position:0% aggressive health checking aggressive 00:14:45.350 --> 00:14:47.379 align:start position:0% aggressive health checking aggressive peer<00:14:45.560> checking<00:14:45.949> load<00:14:46.550> balancing<00:14:47.149> and 00:14:47.379 --> 00:14:47.389 align:start position:0% peer checking load balancing and 00:14:47.389 --> 00:14:51.610 align:start position:0% peer checking load balancing and improper<00:14:47.899> hardware<00:14:49.959> so<00:14:50.959> let's<00:14:51.500> talk<00:14:51.560> about 00:14:51.610 --> 00:14:51.620 align:start position:0% improper hardware so let's talk about 00:14:51.620 --> 00:14:53.139 align:start position:0% improper hardware so let's talk about the<00:14:51.829> infrastructure<00:14:52.189> we<00:14:52.339> actually<00:14:52.639> run<00:14:52.880> in 00:14:53.139 --> 00:14:53.149 align:start position:0% the infrastructure we actually run in 00:14:53.149 --> 00:14:54.910 align:start position:0% the infrastructure we actually run in depth<00:14:53.569> is<00:14:53.839> what<00:14:54.259> we're<00:14:54.350> gonna<00:14:54.470> be<00:14:54.649> open 00:14:54.910 --> 00:14:54.920 align:start position:0% depth is what we're gonna be open 00:14:54.920 --> 00:15:00.910 align:start position:0% depth is what we're gonna be open sourcing<00:14:55.990> so<00:14:58.420> when<00:14:59.420> when<00:15:00.290> we<00:15:00.410> run<00:15:00.680> or<00:15:00.769> knows 00:15:00.910 --> 00:15:00.920 align:start position:0% sourcing so when when we run or knows 00:15:00.920 --> 00:15:04.210 align:start position:0% sourcing so when when we run or knows you<00:15:01.069> run<00:15:01.220> on<00:15:01.310> top<00:15:01.459> of<00:15:01.519> Boone<00:15:02.509> -<00:15:02.689> so<00:15:03.110> we<00:15:03.500> use<00:15:03.800> the 00:15:04.210 --> 00:15:04.220 align:start position:0% you run on top of Boone - so we use the 00:15:04.220 --> 00:15:05.740 align:start position:0% you run on top of Boone - so we use the service<00:15:04.670> construct<00:15:05.240> to<00:15:05.360> start<00:15:05.600> our 00:15:05.740 --> 00:15:05.750 align:start position:0% service construct to start our 00:15:05.750 --> 00:15:08.259 align:start position:0% service construct to start our proprietary<00:15:06.199> tasks<00:15:07.000> here<00:15:08.000> we're<00:15:08.149> actually 00:15:08.259 --> 00:15:08.269 align:start position:0% proprietary tasks here we're actually 00:15:08.269 --> 00:15:10.449 align:start position:0% proprietary tasks here we're actually starting<00:15:08.480> the<00:15:08.750> death<00:15:08.990> process<00:15:09.170> so<00:15:10.160> this<00:15:10.310> is<00:15:10.399> in 00:15:10.449 --> 00:15:10.459 align:start position:0% starting the death process so this is in 00:15:10.459 --> 00:15:11.710 align:start position:0% starting the death process so this is in the<00:15:10.550> open<00:15:10.759> source<00:15:10.850> repo<00:15:11.240> you<00:15:11.360> don't<00:15:11.509> need<00:15:11.630> to 00:15:11.710 --> 00:15:11.720 align:start position:0% the open source repo you don't need to 00:15:11.720 --> 00:15:13.480 align:start position:0% the open source repo you don't need to worry<00:15:11.870> if<00:15:12.050> you<00:15:12.139> can't<00:15:12.319> read<00:15:12.500> the<00:15:12.529> actual<00:15:13.130> code 00:15:13.480 --> 00:15:13.490 align:start position:0% worry if you can't read the actual code 00:15:13.490 --> 00:15:15.519 align:start position:0% worry if you can't read the actual code but<00:15:14.389> let's<00:15:14.420> break<00:15:14.660> down<00:15:14.689> the<00:15:14.839> options<00:15:15.139> we<00:15:15.290> use 00:15:15.519 --> 00:15:15.529 align:start position:0% but let's break down the options we use 00:15:15.529 --> 00:15:17.439 align:start position:0% but let's break down the options we use so<00:15:16.220> we<00:15:16.370> brought<00:15:16.490> the<00:15:16.639> max<00:15:16.790> peers<00:15:17.000> down<00:15:17.149> to<00:15:17.269> fix 00:15:17.439 --> 00:15:17.449 align:start position:0% so we brought the max peers down to fix 00:15:17.449 --> 00:15:20.019 align:start position:0% so we brought the max peers down to fix 50<00:15:18.110> this<00:15:18.680> solved<00:15:19.040> memory<00:15:19.310> leak<00:15:19.490> issue<00:15:19.670> and 00:15:20.019 --> 00:15:20.029 align:start position:0% 50 this solved memory leak issue and 00:15:20.029 --> 00:15:23.019 align:start position:0% 50 this solved memory leak issue and also<00:15:20.540> helped<00:15:20.899> with<00:15:21.110> the<00:15:21.980> data<00:15:22.279> transfer<00:15:22.579> costs 00:15:23.019 --> 00:15:23.029 align:start position:0% also helped with the data transfer costs 00:15:23.029 --> 00:15:24.220 align:start position:0% also helped with the data transfer costs that<00:15:23.149> we'll<00:15:23.269> talk<00:15:23.389> about<00:15:23.449> a<00:15:23.660> couple<00:15:23.930> of<00:15:23.990> slides 00:15:24.220 --> 00:15:24.230 align:start position:0% that we'll talk about a couple of slides 00:15:24.230 --> 00:15:26.829 align:start position:0% that we'll talk about a couple of slides we<00:15:25.040> use<00:15:25.160> a<00:15:25.189> custom<00:15:25.519> data<00:15:25.670> Durr<00:15:25.910> parameter<00:15:26.720> to 00:15:26.829 --> 00:15:26.839 align:start position:0% we use a custom data Durr parameter to 00:15:26.839 --> 00:15:28.449 align:start position:0% we use a custom data Durr parameter to take<00:15:26.959> advantage<00:15:27.259> of<00:15:27.350> the<00:15:27.410> nvme<00:15:27.860> drives<00:15:28.160> that 00:15:28.449 --> 00:15:28.459 align:start position:0% take advantage of the nvme drives that 00:15:28.459 --> 00:15:31.720 align:start position:0% take advantage of the nvme drives that AWS<00:15:28.910> recently<00:15:29.300> made<00:15:29.480> available<00:15:30.550> right<00:15:31.550> now 00:15:31.720 --> 00:15:31.730 align:start position:0% AWS recently made available right now 00:15:31.730 --> 00:15:33.490 align:start position:0% AWS recently made available right now our<00:15:31.910> default<00:15:32.240> is<00:15:32.480> cores<00:15:33.110> turn 00:15:33.490 --> 00:15:33.500 align:start position:0% our default is cores turn 00:15:33.500 --> 00:15:35.860 align:start position:0% our default is cores turn everywhere<00:15:34.010> you<00:15:34.730> may<00:15:34.910> want<00:15:35.210> to<00:15:35.300> restrict<00:15:35.780> this 00:15:35.860 --> 00:15:35.870 align:start position:0% everywhere you may want to restrict this 00:15:35.870 --> 00:15:38.200 align:start position:0% everywhere you may want to restrict this for<00:15:36.080> your<00:15:36.200> own<00:15:36.230> implementation<00:15:37.030> we<00:15:38.030> use<00:15:38.120> a 00:15:38.200 --> 00:15:38.210 align:start position:0% for your own implementation we use a 00:15:38.210 --> 00:15:40.180 align:start position:0% for your own implementation we use a four<00:15:38.390> gig<00:15:38.570> cache<00:15:38.840> to<00:15:39.560> speed<00:15:39.770> up<00:15:39.860> the<00:15:39.920> initial 00:15:40.180 --> 00:15:40.190 align:start position:0% four gig cache to speed up the initial 00:15:40.190 --> 00:15:44.140 align:start position:0% four gig cache to speed up the initial sync<00:15:40.400> process<00:15:42.310> this<00:15:43.310> is<00:15:43.580> the<00:15:43.760> health<00:15:43.970> check 00:15:44.140 --> 00:15:44.150 align:start position:0% sync process this is the health check 00:15:44.150 --> 00:15:45.850 align:start position:0% sync process this is the health check that<00:15:44.300> we<00:15:44.360> use<00:15:44.450> to<00:15:44.570> keep<00:15:44.630> our<00:15:44.780> nodes<00:15:44.960> in<00:15:45.500> sync 00:15:45.850 --> 00:15:45.860 align:start position:0% that we use to keep our nodes in sync 00:15:45.860 --> 00:15:47.950 align:start position:0% that we use to keep our nodes in sync it's<00:15:46.610> probably<00:15:46.940> pretty<00:15:47.090> small<00:15:47.210> but<00:15:47.750> what<00:15:47.870> it 00:15:47.950 --> 00:15:47.960 align:start position:0% it's probably pretty small but what it 00:15:47.960 --> 00:15:50.740 align:start position:0% it's probably pretty small but what it essentially<00:15:48.290> is<00:15:48.560> is<00:15:48.890> a<00:15:49.400> node.js<00:15:49.940> Express<00:15:50.540> app 00:15:50.740 --> 00:15:50.750 align:start position:0% essentially is is a node.js Express app 00:15:50.750 --> 00:15:52.810 align:start position:0% essentially is is a node.js Express app that<00:15:51.500> just<00:15:51.710> does<00:15:51.860> the<00:15:52.010> waterfall<00:15:52.400> queries 00:15:52.810 --> 00:15:52.820 align:start position:0% that just does the waterfall queries 00:15:52.820 --> 00:15:56.470 align:start position:0% that just does the waterfall queries over<00:15:53.150> our<00:15:53.840> peers<00:15:54.380> to<00:15:54.590> determine<00:15:54.980> state<00:15:55.360> so<00:15:56.360> we 00:15:56.470 --> 00:15:56.480 align:start position:0% over our peers to determine state so we 00:15:56.480 --> 00:15:58.900 align:start position:0% over our peers to determine state so we compared<00:15:56.840> to<00:15:57.050> in<00:15:57.200> fira<00:15:57.410> and<00:15:57.590> ether<00:15:57.830> scan<00:15:58.070> but 00:15:58.900 --> 00:15:58.910 align:start position:0% compared to in fira and ether scan but 00:15:58.910 --> 00:16:00.400 align:start position:0% compared to in fira and ether scan but you<00:15:58.970> can<00:15:59.090> certainly<00:15:59.210> customize<00:15:59.750> this<00:15:59.960> to<00:16:00.200> run 00:16:00.400 --> 00:16:00.410 align:start position:0% you can certainly customize this to run 00:16:00.410 --> 00:16:02.830 align:start position:0% you can certainly customize this to run on<00:16:00.650> additional<00:16:01.130> peers<00:16:01.540> we<00:16:02.540> consider 00:16:02.830 --> 00:16:02.840 align:start position:0% on additional peers we consider 00:16:02.840 --> 00:16:04.690 align:start position:0% on additional peers we consider ourselves<00:16:02.960> to<00:16:03.170> be<00:16:03.230> in<00:16:03.440> sync<00:16:03.800> if<00:16:04.070> we're<00:16:04.520> within 00:16:04.690 --> 00:16:04.700 align:start position:0% ourselves to be in sync if we're within 00:16:04.700 --> 00:16:07.030 align:start position:0% ourselves to be in sync if we're within 10<00:16:04.940> blocks<00:16:05.060> of<00:16:05.330> the<00:16:05.420> head<00:16:05.600> and<00:16:05.870> you<00:16:06.710> can<00:16:06.890> make 00:16:07.030 --> 00:16:07.040 align:start position:0% 10 blocks of the head and you can make 00:16:07.040 --> 00:16:08.290 align:start position:0% 10 blocks of the head and you can make this<00:16:07.160> tighter<00:16:07.430> but<00:16:07.670> we<00:16:07.760> found<00:16:08.030> that<00:16:08.150> if<00:16:08.270> you 00:16:08.290 --> 00:16:08.300 align:start position:0% this tighter but we found that if you 00:16:08.300 --> 00:16:11.020 align:start position:0% this tighter but we found that if you tighten<00:16:08.780> it<00:16:08.840> up<00:16:09.050> too<00:16:09.890> close<00:16:10.070> to<00:16:10.580> the<00:16:10.670> actual 00:16:11.020 --> 00:16:11.030 align:start position:0% tighten it up too close to the actual 00:16:11.030 --> 00:16:13.570 align:start position:0% tighten it up too close to the actual head<00:16:11.330> state<00:16:11.630> like<00:16:11.810> 3<00:16:12.080> blocks<00:16:12.320> or<00:16:12.500> less<00:16:12.680> you 00:16:13.570 --> 00:16:13.580 align:start position:0% head state like 3 blocks or less you 00:16:13.580 --> 00:16:15.970 align:start position:0% head state like 3 blocks or less you will<00:16:13.970> occasionally<00:16:14.300> drop<00:16:14.690> entirely<00:16:15.110> out<00:16:15.470> of 00:16:15.970 --> 00:16:15.980 align:start position:0% will occasionally drop entirely out of 00:16:15.980 --> 00:16:17.920 align:start position:0% will occasionally drop entirely out of sync<00:16:16.250> with<00:16:16.310> all<00:16:16.580> your<00:16:16.760> nodes<00:16:17.000> when<00:16:17.600> you<00:16:17.690> get<00:16:17.900> a 00:16:17.920 --> 00:16:17.930 align:start position:0% sync with all your nodes when you get a 00:16:17.930 --> 00:16:19.600 align:start position:0% sync with all your nodes when you get a small<00:16:18.350> chain<00:16:18.620> split<00:16:18.950> at<00:16:19.070> the<00:16:19.280> head<00:16:19.460> which 00:16:19.600 --> 00:16:19.610 align:start position:0% small chain split at the head which 00:16:19.610 --> 00:16:21.940 align:start position:0% small chain split at the head which happens<00:16:19.970> on<00:16:20.060> occasion<00:16:20.560> so<00:16:21.560> we<00:16:21.650> decided<00:16:21.890> to 00:16:21.940 --> 00:16:21.950 align:start position:0% happens on occasion so we decided to 00:16:21.950 --> 00:16:24.160 align:start position:0% happens on occasion so we decided to rather<00:16:22.100> stay<00:16:22.400> in<00:16:22.640> sync<00:16:23.360> all<00:16:23.660> right<00:16:23.810> sorry 00:16:24.160 --> 00:16:24.170 align:start position:0% rather stay in sync all right sorry 00:16:24.170 --> 00:16:26.650 align:start position:0% rather stay in sync all right sorry available<00:16:24.830> then<00:16:25.310> it<00:16:25.490> completely<00:16:25.730> in<00:16:26.060> sync<00:16:26.390> all 00:16:26.650 --> 00:16:26.660 align:start position:0% available then it completely in sync all 00:16:26.660 --> 00:16:29.410 align:start position:0% available then it completely in sync all the<00:16:26.930> time<00:16:27.190> we<00:16:28.190> returned<00:16:28.460> back<00:16:28.550> HTTP<00:16:29.210> status 00:16:29.410 --> 00:16:29.420 align:start position:0% the time we returned back HTTP status 00:16:29.420 --> 00:16:31.660 align:start position:0% the time we returned back HTTP status codes<00:16:29.780> that<00:16:30.290> do<00:16:30.470> correctly<00:16:30.860> identify<00:16:31.430> whether 00:16:31.660 --> 00:16:31.670 align:start position:0% codes that do correctly identify whether 00:16:31.670 --> 00:16:32.830 align:start position:0% codes that do correctly identify whether the<00:16:31.850> service<00:16:32.120> is<00:16:32.240> up<00:16:32.420> or<00:16:32.540> down 00:16:32.830 --> 00:16:32.840 align:start position:0% the service is up or down 00:16:32.840 --> 00:16:35.380 align:start position:0% the service is up or down we<00:16:33.650> also<00:16:33.800> returned<00:16:34.160> back<00:16:34.370> a<00:16:34.640> network<00:16:34.970> timeout 00:16:35.380 --> 00:16:35.390 align:start position:0% we also returned back a network timeout 00:16:35.390 --> 00:16:36.730 align:start position:0% we also returned back a network timeout because<00:16:35.630> we<00:16:35.720> found<00:16:35.870> out<00:16:36.020> occasionally<00:16:36.380> in 00:16:36.730 --> 00:16:36.740 align:start position:0% because we found out occasionally in 00:16:36.740 --> 00:16:40.030 align:start position:0% because we found out occasionally in fira<00:16:37.040> just<00:16:37.430> times<00:16:37.610> out<00:16:37.790> our<00:16:37.880> requests<00:16:39.040> we've 00:16:40.030 --> 00:16:40.040 align:start position:0% fira just times out our requests we've 00:16:40.040 --> 00:16:41.020 align:start position:0% fira just times out our requests we've got<00:16:40.130> a<00:16:40.160> couple<00:16:40.310> of<00:16:40.430> the<00:16:40.520> nifty<00:16:40.730> features 00:16:41.020 --> 00:16:41.030 align:start position:0% got a couple of the nifty features 00:16:41.030 --> 00:16:43.090 align:start position:0% got a couple of the nifty features health<00:16:41.840> check<00:16:42.080> is<00:16:42.110> started<00:16:42.530> as<00:16:42.620> a<00:16:42.650> service 00:16:43.090 --> 00:16:43.100 align:start position:0% health check is started as a service 00:16:43.100 --> 00:16:44.920 align:start position:0% health check is started as a service we've<00:16:43.250> run<00:16:43.400> into<00:16:43.580> issues<00:16:43.670> before<00:16:44.240> where<00:16:44.780> our 00:16:44.920 --> 00:16:44.930 align:start position:0% we've run into issues before where our 00:16:44.930 --> 00:16:47.320 align:start position:0% we've run into issues before where our health<00:16:45.110> check<00:16:45.410> was<00:16:46.130> down<00:16:46.700> but<00:16:47.000> the<00:16:47.060> node<00:16:47.210> was 00:16:47.320 --> 00:16:47.330 align:start position:0% health check was down but the node was 00:16:47.330 --> 00:16:49.870 align:start position:0% health check was down but the node was actually<00:16:47.660> off<00:16:47.810> we<00:16:48.650> have<00:16:48.770> cron<00:16:48.980> jobs<00:16:49.310> that 00:16:49.870 --> 00:16:49.880 align:start position:0% actually off we have cron jobs that 00:16:49.880 --> 00:16:51.940 align:start position:0% actually off we have cron jobs that reset<00:16:50.330> the<00:16:50.540> max<00:16:50.780> open<00:16:50.960> file<00:16:51.260> limit<00:16:51.560> this<00:16:51.770> was 00:16:51.940 --> 00:16:51.950 align:start position:0% reset the max open file limit this was 00:16:51.950 --> 00:16:53.920 align:start position:0% reset the max open file limit this was something<00:16:52.190> that<00:16:52.370> Goeff<00:16:53.060> had<00:16:53.450> an<00:16:53.570> issue<00:16:53.690> with 00:16:53.920 --> 00:16:53.930 align:start position:0% something that Goeff had an issue with 00:16:53.930 --> 00:16:56.440 align:start position:0% something that Goeff had an issue with earlier<00:16:54.380> where<00:16:55.100> it<00:16:55.280> was<00:16:55.430> not<00:16:55.760> respecting<00:16:56.360> the 00:16:56.440 --> 00:16:56.450 align:start position:0% earlier where it was not respecting the 00:16:56.450 --> 00:16:58.000 align:start position:0% earlier where it was not respecting the max<00:16:56.630> open<00:16:56.810> file<00:16:57.140> limit<00:16:57.440> so<00:16:57.620> we<00:16:57.710> just<00:16:57.860> wrote<00:16:57.980> a 00:16:58.000 --> 00:16:58.010 align:start position:0% max open file limit so we just wrote a 00:16:58.010 --> 00:17:00.490 align:start position:0% max open file limit so we just wrote a cron<00:16:58.220> job<00:16:58.250> to<00:16:58.580> fix<00:16:58.760> this<00:16:59.000> and<00:16:59.300> then<00:17:00.230> we<00:17:00.320> wrapped 00:17:00.490 --> 00:17:00.500 align:start position:0% cron job to fix this and then we wrapped 00:17:00.500 --> 00:17:03.070 align:start position:0% cron job to fix this and then we wrapped this<00:17:00.650> all<00:17:00.830> in<00:17:01.070> a<00:17:01.160> cloud<00:17:01.400> init<00:17:01.670> script<00:17:02.030> so<00:17:02.810> it 00:17:03.070 --> 00:17:03.080 align:start position:0% this all in a cloud init script so it 00:17:03.080 --> 00:17:04.570 align:start position:0% this all in a cloud init script so it automatically<00:17:03.590> spins<00:17:03.890> up<00:17:04.040> the<00:17:04.130> boxes 00:17:04.570 --> 00:17:04.580 align:start position:0% automatically spins up the boxes 00:17:04.580 --> 00:17:07.420 align:start position:0% automatically spins up the boxes installs<00:17:05.300> all<00:17:05.510> the<00:17:05.660> required<00:17:06.080> programs<00:17:06.680> death 00:17:07.420 --> 00:17:07.430 align:start position:0% installs all the required programs death 00:17:07.430 --> 00:17:10.480 align:start position:0% installs all the required programs death mounts<00:17:08.150> the<00:17:08.270> nvme<00:17:08.720> drives<00:17:09.050> and<00:17:09.400> starts<00:17:10.400> the 00:17:10.480 --> 00:17:10.490 align:start position:0% mounts the nvme drives and starts the 00:17:10.490 --> 00:17:12.490 align:start position:0% mounts the nvme drives and starts the services<00:17:10.970> so<00:17:11.420> your<00:17:11.510> boxes<00:17:11.900> spin<00:17:12.170> up<00:17:12.290> and<00:17:12.440> they 00:17:12.490 --> 00:17:12.500 align:start position:0% services so your boxes spin up and they 00:17:12.500 --> 00:17:13.990 align:start position:0% services so your boxes spin up and they start<00:17:12.740> syncing<00:17:13.190> you<00:17:13.370> can<00:17:13.520> actually<00:17:13.820> check 00:17:13.990 --> 00:17:14.000 align:start position:0% start syncing you can actually check 00:17:14.000 --> 00:17:15.400 align:start position:0% start syncing you can actually check that<00:17:14.150> in<00:17:14.270> a<00:17:14.360> health<00:17:14.540> check<00:17:14.780> that<00:17:14.930> we<00:17:15.020> publish 00:17:15.400 --> 00:17:15.410 align:start position:0% that in a health check that we publish 00:17:15.410 --> 00:17:17.230 align:start position:0% that in a health check that we publish as<00:17:16.010> to<00:17:16.220> whether<00:17:16.370> or<00:17:16.430> not<00:17:16.520> your<00:17:16.760> node<00:17:17.090> is 00:17:17.230 --> 00:17:17.240 align:start position:0% as to whether or not your node is 00:17:17.240 --> 00:17:20.590 align:start position:0% as to whether or not your node is running<00:17:18.790> one<00:17:19.790> of<00:17:19.850> the<00:17:19.940> biggest<00:17:20.150> takeaways<00:17:20.210> we 00:17:20.590 --> 00:17:20.600 align:start position:0% running one of the biggest takeaways we 00:17:20.600 --> 00:17:22.480 align:start position:0% running one of the biggest takeaways we had<00:17:20.720> was<00:17:20.900> the<00:17:20.990> hardware<00:17:21.200> selection<00:17:21.800> that<00:17:22.340> we 00:17:22.480 --> 00:17:22.490 align:start position:0% had was the hardware selection that we 00:17:22.490 --> 00:17:24.370 align:start position:0% had was the hardware selection that we used<00:17:22.760> so<00:17:23.300> we're<00:17:23.390> gonna<00:17:23.480> talk<00:17:23.630> about<00:17:23.660> IO<00:17:24.020> right 00:17:24.370 --> 00:17:24.380 align:start position:0% used so we're gonna talk about IO right 00:17:24.380 --> 00:17:26.500 align:start position:0% used so we're gonna talk about IO right now<00:17:24.670> magnetic<00:17:25.670> and<00:17:25.790> spinning<00:17:26.030> disk<00:17:26.210> drives 00:17:26.500 --> 00:17:26.510 align:start position:0% now magnetic and spinning disk drives 00:17:26.510 --> 00:17:28.780 align:start position:0% now magnetic and spinning disk drives just<00:17:26.840> don't<00:17:27.110> work<00:17:27.350> you<00:17:28.250> just<00:17:28.370> don't<00:17:28.520> stay<00:17:28.700> in 00:17:28.780 --> 00:17:28.790 align:start position:0% just don't work you just don't stay in 00:17:28.790 --> 00:17:31.750 align:start position:0% just don't work you just don't stay in sync<00:17:29.030> you<00:17:29.270> become<00:17:29.510> IO<00:17:29.720> bound<00:17:30.110> and<00:17:30.440> the<00:17:31.430> the 00:17:31.750 --> 00:17:31.760 align:start position:0% sync you become IO bound and the the 00:17:31.760 --> 00:17:34.810 align:start position:0% sync you become IO bound and the the node<00:17:31.940> will<00:17:32.540> freeze<00:17:33.280> similarly<00:17:34.280> provision<00:17:34.670> die 00:17:34.810 --> 00:17:34.820 align:start position:0% node will freeze similarly provision die 00:17:34.820 --> 00:17:36.940 align:start position:0% node will freeze similarly provision die ops<00:17:35.030> is<00:17:35.240> actually<00:17:35.570> not<00:17:35.660> cost<00:17:35.960> effective<00:17:36.110> you 00:17:36.940 --> 00:17:36.950 align:start position:0% ops is actually not cost effective you 00:17:36.950 --> 00:17:38.530 align:start position:0% ops is actually not cost effective you over<00:17:37.220> allocate<00:17:37.610> your<00:17:37.700> I<00:17:37.760> UPS<00:17:38.060> and<00:17:38.330> you're<00:17:38.420> only 00:17:38.530 --> 00:17:38.540 align:start position:0% over allocate your I UPS and you're only 00:17:38.540 --> 00:17:39.910 align:start position:0% over allocate your I UPS and you're only using<00:17:38.750> your<00:17:39.080> max<00:17:39.260> I<00:17:39.440> it's<00:17:39.620> about<00:17:39.680> three 00:17:39.910 --> 00:17:39.920 align:start position:0% using your max I it's about three 00:17:39.920 --> 00:17:41.890 align:start position:0% using your max I it's about three percent<00:17:40.100> of<00:17:40.310> the<00:17:40.400> time<00:17:40.610> so<00:17:41.390> this<00:17:41.510> means<00:17:41.720> you're 00:17:41.890 --> 00:17:41.900 align:start position:0% percent of the time so this means you're 00:17:41.900 --> 00:17:44.290 align:start position:0% percent of the time so this means you're just<00:17:42.050> wasting<00:17:42.380> money<00:17:42.560> so<00:17:43.310> we<00:17:43.460> try<00:17:43.760> moving<00:17:44.090> back 00:17:44.290 --> 00:17:44.300 align:start position:0% just wasting money so we try moving back 00:17:44.300 --> 00:17:47.240 align:start position:0% just wasting money so we try moving back to<00:17:44.570> births<00:17:44.900> based<00:17:45.170> EBS<00:17:46.100> drives 00:17:47.240 --> 00:17:47.250 align:start position:0% to births based EBS drives 00:17:47.250 --> 00:17:48.740 align:start position:0% to births based EBS drives and<00:17:47.340> you<00:17:47.580> have<00:17:47.730> to<00:17:47.850> actually<00:17:48.000> allocate<00:17:48.270> three 00:17:48.740 --> 00:17:48.750 align:start position:0% and you have to actually allocate three 00:17:48.750 --> 00:17:50.180 align:start position:0% and you have to actually allocate three terabytes<00:17:49.169> of<00:17:49.289> storage<00:17:49.350> space<00:17:49.860> to<00:17:50.039> get<00:17:50.130> the 00:17:50.180 --> 00:17:50.190 align:start position:0% terabytes of storage space to get the 00:17:50.190 --> 00:17:52.399 align:start position:0% terabytes of storage space to get the provisioned<00:17:50.640> I<00:17:50.760> ops<00:17:50.940> that<00:17:51.059> you<00:17:51.240> need<00:17:51.480> but 00:17:52.399 --> 00:17:52.409 align:start position:0% provisioned I ops that you need but 00:17:52.409 --> 00:17:54.830 align:start position:0% provisioned I ops that you need but again<00:17:52.740> because<00:17:53.580> now<00:17:53.940> you're<00:17:54.150> below<00:17:54.570> where 00:17:54.830 --> 00:17:54.840 align:start position:0% again because now you're below where 00:17:54.840 --> 00:17:56.149 align:start position:0% again because now you're below where you'd<00:17:54.990> want<00:17:55.049> to<00:17:55.289> be<00:17:55.409> in<00:17:55.500> AI<00:17:55.650> ops<00:17:55.919> you're 00:17:56.149 --> 00:17:56.159 align:start position:0% you'd want to be in AI ops you're 00:17:56.159 --> 00:17:57.950 align:start position:0% you'd want to be in AI ops you're actually<00:17:56.520> hitting<00:17:56.669> caps<00:17:57.150> when<00:17:57.450> a<00:17:57.480> large<00:17:57.690> block 00:17:57.950 --> 00:17:57.960 align:start position:0% actually hitting caps when a large block 00:17:57.960 --> 00:18:00.380 align:start position:0% actually hitting caps when a large block comes<00:17:58.260> in<00:17:58.470> and<00:17:58.710> so<00:17:59.100> now<00:17:59.220> you<00:17:59.280> have<00:17:59.580> a<00:17:59.610> right<00:17:59.970> to 00:18:00.380 --> 00:18:00.390 align:start position:0% comes in and so now you have a right to 00:18:00.390 --> 00:18:02.600 align:start position:0% comes in and so now you have a right to that<00:18:00.690> will<00:18:01.380> build<00:18:01.590> up<00:18:01.770> over<00:18:01.980> time<00:18:02.100> you<00:18:02.400> know<00:18:02.490> it 00:18:02.600 --> 00:18:02.610 align:start position:0% that will build up over time you know it 00:18:02.610 --> 00:18:05.390 align:start position:0% that will build up over time you know it falls<00:18:02.820> out<00:18:02.970> of<00:18:03.090> sync<00:18:03.559> EFS<00:18:04.559> is<00:18:04.770> something<00:18:05.130> that 00:18:05.390 --> 00:18:05.400 align:start position:0% falls out of sync EFS is something that 00:18:05.400 --> 00:18:07.010 align:start position:0% falls out of sync EFS is something that AWS<00:18:05.909> provides<00:18:05.940> an<00:18:06.480> infinitely<00:18:06.840> scalable 00:18:07.010 --> 00:18:07.020 align:start position:0% AWS provides an infinitely scalable 00:18:07.020 --> 00:18:10.399 align:start position:0% AWS provides an infinitely scalable network<00:18:07.530> attached<00:18:07.799> storage<00:18:08.429> and<00:18:09.080> the<00:18:10.080> network 00:18:10.399 --> 00:18:10.409 align:start position:0% network attached storage and the network 00:18:10.409 --> 00:18:12.200 align:start position:0% network attached storage and the network overhead<00:18:10.650> actually<00:18:11.250> makes<00:18:11.400> it<00:18:11.520> so<00:18:11.669> your<00:18:11.940> node 00:18:12.200 --> 00:18:12.210 align:start position:0% overhead actually makes it so your node 00:18:12.210 --> 00:18:13.549 align:start position:0% overhead actually makes it so your node will<00:18:12.390> never<00:18:12.539> sink<00:18:12.840> as<00:18:12.960> well<00:18:13.080> so<00:18:13.260> just<00:18:13.320> don't 00:18:13.549 --> 00:18:13.559 align:start position:0% will never sink as well so just don't 00:18:13.559 --> 00:18:16.340 align:start position:0% will never sink as well so just don't try<00:18:13.740> that<00:18:13.770> and<00:18:14.690> nvme<00:18:15.690> drives<00:18:15.929> were<00:18:16.080> announced 00:18:16.340 --> 00:18:16.350 align:start position:0% try that and nvme drives were announced 00:18:16.350 --> 00:18:18.289 align:start position:0% try that and nvme drives were announced a<00:18:16.409> few<00:18:16.559> months<00:18:16.799> ago<00:18:16.919> that's<00:18:17.640> actually<00:18:18.059> what<00:18:18.179> we 00:18:18.289 --> 00:18:18.299 align:start position:0% a few months ago that's actually what we 00:18:18.299 --> 00:18:22.909 align:start position:0% a few months ago that's actually what we use<00:18:18.330> it<00:18:18.929> is<00:18:19.200> a<00:18:19.590> very<00:18:20.130> good<00:18:20.490> io<00:18:21.320> based<00:18:22.320> instance 00:18:22.909 --> 00:18:22.919 align:start position:0% use it is a very good io based instance 00:18:22.919 --> 00:18:27.740 align:start position:0% use it is a very good io based instance store<00:18:23.159> drive<00:18:24.830> familywize<00:18:26.000> death<00:18:27.000> is<00:18:27.179> io 00:18:27.740 --> 00:18:27.750 align:start position:0% store drive familywize death is io 00:18:27.750 --> 00:18:29.600 align:start position:0% store drive familywize death is io bounded<00:18:28.289> incoming<00:18:28.679> data<00:18:28.860> and<00:18:29.100> compute<00:18:29.490> bound 00:18:29.600 --> 00:18:29.610 align:start position:0% bounded incoming data and compute bound 00:18:29.610 --> 00:18:32.690 align:start position:0% bounded incoming data and compute bound and<00:18:29.789> serving<00:18:30.059> requests<00:18:30.630> so<00:18:31.590> m4<00:18:32.100> and<00:18:32.220> m5 00:18:32.690 --> 00:18:32.700 align:start position:0% and serving requests so m4 and m5 00:18:32.700 --> 00:18:35.029 align:start position:0% and serving requests so m4 and m5 general<00:18:33.419> classes<00:18:33.840> over<00:18:33.990> allocate<00:18:34.440> memory<00:18:34.590> we 00:18:35.029 --> 00:18:35.039 align:start position:0% general classes over allocate memory we 00:18:35.039 --> 00:18:38.120 align:start position:0% general classes over allocate memory we don't<00:18:35.220> use<00:18:35.429> those<00:18:35.960> any<00:18:36.960> other<00:18:37.200> memory<00:18:37.919> heavy 00:18:38.120 --> 00:18:38.130 align:start position:0% don't use those any other memory heavy 00:18:38.130 --> 00:18:41.539 align:start position:0% don't use those any other memory heavy family<00:18:39.000> is<00:18:39.179> actually<00:18:39.390> not<00:18:39.870> going<00:18:40.140> to<00:18:40.260> be<00:18:40.549> cost 00:18:41.539 --> 00:18:41.549 align:start position:0% family is actually not going to be cost 00:18:41.549 --> 00:18:44.690 align:start position:0% family is actually not going to be cost effective<00:18:41.970> I<00:18:42.260> three<00:18:43.260> classes<00:18:43.770> is<00:18:44.039> actually 00:18:44.690 --> 00:18:44.700 align:start position:0% effective I three classes is actually 00:18:44.700 --> 00:18:47.060 align:start position:0% effective I three classes is actually what<00:18:44.820> inferior<00:18:45.240> uses<00:18:45.630> or<00:18:46.470> use<00:18:46.679> the<00:18:46.890> time<00:18:47.039> I 00:18:47.060 --> 00:18:47.070 align:start position:0% what inferior uses or use the time I 00:18:47.070 --> 00:18:49.370 align:start position:0% what inferior uses or use the time I talk<00:18:47.220> to<00:18:47.490> them<00:18:47.610> so<00:18:48.120> they<00:18:48.240> work<00:18:48.390> well<00:18:48.659> but<00:18:48.960> nvm 00:18:49.370 --> 00:18:49.380 align:start position:0% talk to them so they work well but nvm 00:18:49.380 --> 00:18:51.230 align:start position:0% talk to them so they work well but nvm you<00:18:49.409> drives<00:18:49.710> are<00:18:49.890> now<00:18:50.010> back<00:18:50.280> out<00:18:50.549> and<00:18:50.850> that's<00:18:51.120> a 00:18:51.230 --> 00:18:51.240 align:start position:0% you drives are now back out and that's a 00:18:51.240 --> 00:18:53.830 align:start position:0% you drives are now back out and that's a better<00:18:51.600> instance<00:18:52.080> store<00:18:52.380> than<00:18:52.980> the<00:18:53.190> i3<00:18:53.549> class 00:18:53.830 --> 00:18:53.840 align:start position:0% better instance store than the i3 class 00:18:53.840 --> 00:18:57.470 align:start position:0% better instance store than the i3 class we<00:18:54.840> run<00:18:55.020> C<00:18:55.260> 5<00:18:55.440> D<00:18:55.620> is<00:18:55.679> in<00:18:55.830> production<00:18:56.330> this<00:18:57.330> is 00:18:57.470 --> 00:18:57.480 align:start position:0% we run C 5 D is in production this is 00:18:57.480 --> 00:19:00.250 align:start position:0% we run C 5 D is in production this is the<00:18:57.600> proper<00:18:58.080> mix<00:18:58.860> of<00:18:59.010> compute<00:18:59.280> power<00:18:59.309> with<00:18:59.880> IO 00:19:00.250 --> 00:19:00.260 align:start position:0% the proper mix of compute power with IO 00:19:00.260 --> 00:19:03.830 align:start position:0% the proper mix of compute power with IO 4<00:19:01.260> what<00:19:01.500> we<00:19:01.679> what<00:19:02.520> we<00:19:02.549> run<00:19:02.940> we<00:19:03.419> found<00:19:03.659> anything 00:19:03.830 --> 00:19:03.840 align:start position:0% 4 what we what we run we found anything 00:19:03.840 --> 00:19:05.659 align:start position:0% 4 what we what we run we found anything smaller<00:19:04.380> than<00:19:04.500> a<00:19:04.620> large<00:19:04.890> instance<00:19:05.460> is 00:19:05.659 --> 00:19:05.669 align:start position:0% smaller than a large instance is 00:19:05.669 --> 00:19:06.649 align:start position:0% smaller than a large instance is actually<00:19:06.030> not<00:19:06.150> going<00:19:06.270> to<00:19:06.360> give<00:19:06.450> you<00:19:06.510> enough 00:19:06.649 --> 00:19:06.659 align:start position:0% actually not going to give you enough 00:19:06.659 --> 00:19:08.659 align:start position:0% actually not going to give you enough power<00:19:06.900> to<00:19:07.200> run<00:19:07.710> your<00:19:07.860> nodes<00:19:08.039> in<00:19:08.220> production<00:19:08.580> at 00:19:08.659 --> 00:19:08.669 align:start position:0% power to run your nodes in production at 00:19:08.669 --> 00:19:11.330 align:start position:0% power to run your nodes in production at scale<00:19:08.970> and<00:19:09.240> any<00:19:10.110> other<00:19:10.289> family<00:19:10.500> than<00:19:10.770> these<00:19:10.890> it 00:19:11.330 --> 00:19:11.340 align:start position:0% scale and any other family than these it 00:19:11.340 --> 00:19:16.580 align:start position:0% scale and any other family than these it it's<00:19:11.820> just<00:19:12.140> cost<00:19:13.140> too<00:19:13.289> much<00:19:13.789> so<00:19:14.789> we<00:19:14.970> run<00:19:15.539> 6<00:19:16.200> c<00:19:16.409> 5 00:19:16.580 --> 00:19:16.590 align:start position:0% it's just cost too much so we run 6 c 5 00:19:16.590 --> 00:19:19.010 align:start position:0% it's just cost too much so we run 6 c 5 d<00:19:16.770> 4<00:19:17.010> excels<00:19:17.340> in<00:19:17.460> production<00:19:17.880> and<00:19:18.240> we<00:19:18.299> have<00:19:18.900> not 00:19:19.010 --> 00:19:19.020 align:start position:0% d 4 excels in production and we have not 00:19:19.020 --> 00:19:20.659 align:start position:0% d 4 excels in production and we have not had<00:19:19.230> a<00:19:19.289> single<00:19:19.500> instance<00:19:19.950> the<00:19:20.039> downtime<00:19:20.370> since 00:19:20.659 --> 00:19:20.669 align:start position:0% had a single instance the downtime since 00:19:20.669 --> 00:19:24.830 align:start position:0% had a single instance the downtime since August<00:19:21.090> last<00:19:23.120> factors<00:19:24.120> cost<00:19:24.390> so<00:19:24.600> what<00:19:24.720> do<00:19:24.780> you 00:19:24.830 --> 00:19:24.840 align:start position:0% August last factors cost so what do you 00:19:24.840 --> 00:19:26.390 align:start position:0% August last factors cost so what do you pay<00:19:24.960> for<00:19:24.990> you<00:19:25.230> pay<00:19:25.350> for<00:19:25.409> the<00:19:25.710> ec2<00:19:26.010> instances 00:19:26.390 --> 00:19:26.400 align:start position:0% pay for you pay for the ec2 instances 00:19:26.400 --> 00:19:29.029 align:start position:0% pay for you pay for the ec2 instances pay<00:19:27.179> for<00:19:27.330> the<00:19:27.390> load<00:19:27.539> balancers<00:19:28.110> and<00:19:28.350> then<00:19:28.950> when 00:19:29.029 --> 00:19:29.039 align:start position:0% pay for the load balancers and then when 00:19:29.039 --> 00:19:30.529 align:start position:0% pay for the load balancers and then when you<00:19:29.100> also<00:19:29.250> pay<00:19:29.429> for<00:19:29.460> outbound<00:19:29.820> data<00:19:30.090> transfer 00:19:30.529 --> 00:19:30.539 align:start position:0% you also pay for outbound data transfer 00:19:30.539 --> 00:19:32.330 align:start position:0% you also pay for outbound data transfer something<00:19:31.200> we<00:19:31.380> didn't<00:19:31.710> know<00:19:31.890> when<00:19:32.070> we<00:19:32.159> had<00:19:32.309> our 00:19:32.330 --> 00:19:32.340 align:start position:0% something we didn't know when we had our 00:19:32.340 --> 00:19:34.220 align:start position:0% something we didn't know when we had our max<00:19:32.580> pure<00:19:32.789> set<00:19:32.970> as<00:19:33.090> high<00:19:33.240> as<00:19:33.390> they<00:19:33.539> were<00:19:33.570> was 00:19:34.220 --> 00:19:34.230 align:start position:0% max pure set as high as they were was 00:19:34.230 --> 00:19:35.299 align:start position:0% max pure set as high as they were was your<00:19:34.350> outbound<00:19:34.590> data<00:19:34.799> transfer<00:19:35.190> actually 00:19:35.299 --> 00:19:35.309 align:start position:0% your outbound data transfer actually 00:19:35.309 --> 00:19:36.560 align:start position:0% your outbound data transfer actually runs<00:19:35.580> into<00:19:35.760> the<00:19:35.820> thousands<00:19:36.179> of<00:19:36.270> dollars<00:19:36.330> per 00:19:36.560 --> 00:19:36.570 align:start position:0% runs into the thousands of dollars per 00:19:36.570 --> 00:19:38.810 align:start position:0% runs into the thousands of dollars per month<00:19:36.750> so<00:19:37.590> we've<00:19:37.770> dropped<00:19:37.980> this<00:19:38.159> down<00:19:38.370> with<00:19:38.520> 50 00:19:38.810 --> 00:19:38.820 align:start position:0% month so we've dropped this down with 50 00:19:38.820 --> 00:19:42.080 align:start position:0% month so we've dropped this down with 50 peers<00:19:39.240> we<00:19:40.049> actually<00:19:40.470> end<00:19:40.620> up<00:19:40.770> with<00:19:41.100> about<00:19:41.340> 2<00:19:42.000> to 00:19:42.080 --> 00:19:42.090 align:start position:0% peers we actually end up with about 2 to 00:19:42.090 --> 00:19:45.590 align:start position:0% peers we actually end up with about 2 to 3<00:19:42.240> terabytes<00:19:42.690> out<00:19:42.900> every<00:19:43.320> month<00:19:44.390> real<00:19:45.390> quick 00:19:45.590 --> 00:19:45.600 align:start position:0% 3 terabytes out every month real quick 00:19:45.600 --> 00:19:49.940 align:start position:0% 3 terabytes out every month real quick slide<00:19:46.880> why<00:19:47.880> not<00:19:48.030> parody<00:19:48.360> when<00:19:49.169> we<00:19:49.320> did<00:19:49.530> parody 00:19:49.940 --> 00:19:49.950 align:start position:0% slide why not parody when we did parody 00:19:49.950 --> 00:19:51.680 align:start position:0% slide why not parody when we did parody about<00:19:50.370> 5<00:19:50.640> months<00:19:50.789> ago<00:19:51.059> we<00:19:51.299> were<00:19:51.330> getting<00:19:51.450> a 00:19:51.680 --> 00:19:51.690 align:start position:0% about 5 months ago we were getting a 00:19:51.690 --> 00:19:53.990 align:start position:0% about 5 months ago we were getting a bunch<00:19:51.870> of<00:19:51.990> false<00:19:52.230> nodes<00:19:53.070> so<00:19:53.669> we<00:19:53.700> were<00:19:53.909> not 00:19:53.990 --> 00:19:54.000 align:start position:0% bunch of false nodes so we were not 00:19:54.000 --> 00:19:55.789 align:start position:0% bunch of false nodes so we were not actually<00:19:54.390> staying<00:19:54.630> in<00:19:54.840> sync<00:19:55.169> appropriately 00:19:55.789 --> 00:19:55.799 align:start position:0% actually staying in sync appropriately 00:19:55.799 --> 00:19:57.529 align:start position:0% actually staying in sync appropriately with<00:19:55.919> enough<00:19:56.130> peers<00:19:56.399> so<00:19:57.059> we<00:19:57.179> just<00:19:57.330> dropped 00:19:57.529 --> 00:19:57.539 align:start position:0% with enough peers so we just dropped 00:19:57.539 --> 00:19:59.299 align:start position:0% with enough peers so we just dropped them<00:19:57.690> but<00:19:58.320> version<00:19:58.710> 2.0 00:19:59.299 --> 00:19:59.309 align:start position:0% them but version 2.0 00:19:59.309 --> 00:20:00.910 align:start position:0% them but version 2.0 we<00:19:59.580> certainly<00:19:59.789> want<00:20:00.030> to<00:20:00.120> go<00:20:00.240> ahead<00:20:00.360> and 00:20:00.910 --> 00:20:00.920 align:start position:0% we certainly want to go ahead and 00:20:00.920 --> 00:20:02.260 align:start position:0% we certainly want to go ahead and looking<00:20:01.160> at<00:20:01.220> other<00:20:01.370> clients<00:20:01.790> to<00:20:01.820> bring<00:20:02.090> out 00:20:02.260 --> 00:20:02.270 align:start position:0% looking at other clients to bring out 00:20:02.270 --> 00:20:03.880 align:start position:0% looking at other clients to bring out and<00:20:02.480> I<00:20:02.870> think<00:20:02.990> that's<00:20:03.140> something<00:20:03.290> that<00:20:03.410> we<00:20:03.560> you 00:20:03.880 --> 00:20:03.890 align:start position:0% and I think that's something that we you 00:20:03.890 --> 00:20:05.020 align:start position:0% and I think that's something that we you know<00:20:03.980> kind<00:20:04.220> of<00:20:04.280> call<00:20:04.460> for<00:20:04.490> help<00:20:04.700> on<00:20:04.910> the 00:20:05.020 --> 00:20:05.030 align:start position:0% know kind of call for help on the 00:20:05.030 --> 00:20:06.880 align:start position:0% know kind of call for help on the open-source<00:20:05.360> side<00:20:05.630> if<00:20:06.260> you<00:20:06.350> guys<00:20:06.440> know<00:20:06.680> how<00:20:06.860> to 00:20:06.880 --> 00:20:06.890 align:start position:0% open-source side if you guys know how to 00:20:06.890 --> 00:20:09.010 align:start position:0% open-source side if you guys know how to properly<00:20:07.430> configure<00:20:07.640> parity<00:20:08.150> like<00:20:08.810> this 00:20:09.010 --> 00:20:09.020 align:start position:0% properly configure parity like this 00:20:09.020 --> 00:20:11.340 align:start position:0% properly configure parity like this month<00:20:09.230> that<00:20:09.500> would<00:20:09.620> be<00:20:09.710> great 00:20:11.340 --> 00:20:11.350 align:start position:0% month that would be great 00:20:11.350 --> 00:20:13.210 align:start position:0% month that would be great and<00:20:12.350> lastly<00:20:12.650> here's<00:20:12.830> the<00:20:12.920> CloudFormation 00:20:13.210 --> 00:20:13.220 align:start position:0% and lastly here's the CloudFormation 00:20:13.220 --> 00:20:16.270 align:start position:0% and lastly here's the CloudFormation stack<00:20:13.700> like<00:20:13.970> I<00:20:14.060> said<00:20:14.210> we<00:20:14.330> do<00:20:14.540> run<00:20:14.690> an<00:20:14.840> AWS<00:20:15.380> but 00:20:16.270 --> 00:20:16.280 align:start position:0% stack like I said we do run an AWS but 00:20:16.280 --> 00:20:18.340 align:start position:0% stack like I said we do run an AWS but this<00:20:16.400> URL<00:20:16.700> will<00:20:16.910> take<00:20:17.090> you<00:20:17.330> basically<00:20:17.780> to<00:20:18.020> our 00:20:18.340 --> 00:20:18.350 align:start position:0% this URL will take you basically to our 00:20:18.350 --> 00:20:20.290 align:start position:0% this URL will take you basically to our github<00:20:18.560> repository<00:20:18.790> where<00:20:19.790> you<00:20:19.970> can<00:20:20.090> go<00:20:20.180> ahead 00:20:20.290 --> 00:20:20.300 align:start position:0% github repository where you can go ahead 00:20:20.300 --> 00:20:22.420 align:start position:0% github repository where you can go ahead and<00:20:20.750> click<00:20:21.230> with<00:20:21.410> a<00:20:21.470> single<00:20:21.890> launch<00:20:22.070> stack 00:20:22.420 --> 00:20:22.430 align:start position:0% and click with a single launch stack 00:20:22.430 --> 00:20:24.490 align:start position:0% and click with a single launch stack spin<00:20:23.300> up<00:20:23.390> the<00:20:23.420> exact<00:20:23.720> stack<00:20:23.990> that<00:20:24.110> we<00:20:24.170> run<00:20:24.410> in 00:20:24.490 --> 00:20:24.500 align:start position:0% spin up the exact stack that we run in 00:20:24.500 --> 00:20:26.530 align:start position:0% spin up the exact stack that we run in production<00:20:25.000> that's<00:20:26.000> all<00:20:26.150> I<00:20:26.210> have 00:20:26.530 --> 00:20:26.540 align:start position:0% production that's all I have 00:20:26.540 --> 00:20:28.400 align:start position:0% production that's all I have thanks<00:20:26.870> guys 00:20:28.400 --> 00:20:28.410 align:start position:0% thanks guys 00:20:28.410 --> 00:20:39.970 align:start position:0% thanks guys [Applause] 00:20:39.970 --> 00:20:39.980 align:start position:0% 00:20:39.980 --> 00:20:42.040 align:start position:0% you