Scaling up scrapers with Prefect and Google Cloud

By • • Prefect Live in Twitch

Recording

Show the timestamped transcript
  1. [Music]
  2. [Music]
  3. [Music]
  4. [Music]
  5. [Music]
  6. [Music]
  7. [Music]
  8. [Music]
  9. [Music]
  10. hey everybody welcome back to another
  11. hey everybody welcome back to another
  12. hey everybody welcome back to another prefect live i'm chris reuter and today
  13. prefect live i'm chris reuter and today
  14. prefect live i'm chris reuter and today i'm here with ben welsh so hey ben how's
  15. i'm here with ben welsh so hey ben how's
  16. i'm here with ben welsh so hey ben how's it going
  17. it going
  18. it going good thanks for having me absolutely so
  19. good thanks for having me absolutely so
  20. good thanks for having me absolutely so ben is an l.a times journalist and this
  21. ben is an l.a times journalist and this
  22. ben is an l.a times journalist and this is really cool john s knight journalism
  23. is really cool john s knight journalism
  24. is really cool john s knight journalism fellow at stanford i see you've got the
  25. fellow at stanford i see you've got the
  26. fellow at stanford i see you've got the hat on today he's been spending they're
  27. hat on today he's been spending they're
  28. hat on today he's been spending they're spending the year working on the big
  29. spending the year working on the big
  30. spending the year working on the big look local news project so first thing
  31. look local news project so first thing
  32. look local news project so first thing i'd love to hear what is the big local
  33. i'd love to hear what is the big local
  34. i'd love to hear what is the big local news project what are you working on
  35. news project what are you working on
  36. news project what are you working on sure big local news is a program at the
  37. sure big local news is a program at the
  38. sure big local news is a program at the stanford university journalism school
  39. stanford university journalism school
  40. stanford university journalism school that's looking to kind of become
  41. that's looking to kind of become
  42. that's looking to kind of become a hub
  43. a hub
  44. a hub for
  45. for
  46. for gathering uh analyzing organizing uh
  47. gathering uh analyzing organizing uh
  48. gathering uh analyzing organizing uh large data sets for use in local news
  49. large data sets for use in local news
  50. large data sets for use in local news so you know
  51. so you know
  52. so you know whether you know it or not there are
  53. whether you know it or not there are
  54. whether you know it or not there are computer programmers in newsrooms all
  55. computer programmers in newsrooms all
  56. computer programmers in newsrooms all across america who are using their kind
  57. across america who are using their kind
  58. across america who are using their kind of skills to grab data wrestle it down
  59. of skills to grab data wrestle it down
  60. of skills to grab data wrestle it down make sense of it and turn it into
  61. make sense of it and turn it into
  62. make sense of it and turn it into stories to find and tell stories and the
  63. stories to find and tell stories and the
  64. stories to find and tell stories and the goal of the program at stanford is to to
  65. goal of the program at stanford is to to
  66. goal of the program at stanford is to to help newsrooms make that happen
  67. help newsrooms make that happen
  68. help newsrooms make that happen that uh data journalism concept i think
  69. that uh data journalism concept i think
  70. that uh data journalism concept i think one was definitely new to me and you you
  71. one was definitely new to me and you you
  72. one was definitely new to me and you you joined the slack you started asking
  73. joined the slack you started asking
  74. joined the slack you started asking questions and i really appreciate your
  75. questions and i really appreciate your
  76. questions and i really appreciate your asking some some interesting questions
  77. asking some some interesting questions
  78. asking some some interesting questions that i think will help other people get
  79. that i think will help other people get
  80. that i think will help other people get started too so you've already been a
  81. started too so you've already been a
  82. started too so you've already been a really helpful valuable member of the
  83. really helpful valuable member of the
  84. really helpful valuable member of the community i really want to thank you for
  85. community i really want to thank you for
  86. community i really want to thank you for that too
  87. that too
  88. that too ben was telling me before he came on
  89. ben was telling me before he came on
  90. ben was telling me before he came on he's originally from iowa so hard
  91. he's originally from iowa so hard
  92. he's originally from iowa so hard question today
  93. question today
  94. question today uh this is the i to me the i o question
  95. uh this is the i to me the i o question
  96. uh this is the i to me the i o question uh cyclones or hawkeyes oh it's
  97. uh cyclones or hawkeyes oh it's
  98. uh cyclones or hawkeyes oh it's definitely the iowa hawkeyes i grew up
  99. definitely the iowa hawkeyes i grew up
  100. definitely the iowa hawkeyes i grew up near iowa city iowa and a big hawkeye
  101. near iowa city iowa and a big hawkeye
  102. near iowa city iowa and a big hawkeye football fan and i go back for a game
  103. football fan and i go back for a game
  104. football fan and i go back for a game every year i was there for penn state
  105. every year i was there for penn state
  106. every year i was there for penn state last fall that's awesome
  107. last fall that's awesome
  108. last fall that's awesome so today we're talking about the work
  109. so today we're talking about the work
  110. so today we're talking about the work you've been doing has been around web
  111. you've been doing has been around web
  112. you've been doing has been around web scraping i think was really probably
  113. scraping i think was really probably
  114. scraping i think was really probably more relevant to a lot of the data
  115. more relevant to a lot of the data
  116. more relevant to a lot of the data journalists out there but i think some
  117. journalists out there but i think some
  118. journalists out there but i think some of our other community members could
  119. of our other community members could
  120. of our other community members could definitely learn a little bit from your
  121. definitely learn a little bit from your
  122. definitely learn a little bit from your use case so i know you're on gcp um do
  123. use case so i know you're on gcp um do
  124. use case so i know you're on gcp um do you want to give a brief overview of
  125. you want to give a brief overview of
  126. you want to give a brief overview of what we're going to talk about today and
  127. what we're going to talk about today and
  128. what we're going to talk about today and then we can get right into it
  129. then we can get right into it
  130. then we can get right into it yeah i was thinking i could give some
  131. yeah i was thinking i could give some
  132. yeah i was thinking i could give some examples of the type of work that our
  133. examples of the type of work that our
  134. examples of the type of work that our team does to turn data into stories into
  135. team does to turn data into stories into
  136. team does to turn data into stories into news and then we can really dig into one
  137. news and then we can really dig into one
  138. news and then we can really dig into one of kind of our
  139. of kind of our
  140. of kind of our new projects still at an early stage
  141. new projects still at an early stage
  142. new projects still at an early stage that's trying to take advantage of
  143. that's trying to take advantage of
  144. that's trying to take advantage of prefect to really scale up what we can
  145. prefect to really scale up what we can
  146. prefect to really scale up what we can pull off that sounds awesome cool so if
  147. pull off that sounds awesome cool so if
  148. pull off that sounds awesome cool so if you're ready i'll switch over to your
  149. you're ready i'll switch over to your
  150. you're ready i'll switch over to your screen
  151. screen
  152. screen yeah sure can you see it yep it's all
  153. yeah sure can you see it yep it's all
  154. yeah sure can you see it yep it's all good
  155. good
  156. good great so we're starting off here right
  157. great so we're starting off here right
  158. great so we're starting off here right now
  159. now
  160. now with a story that our team put out just
  161. with a story that our team put out just
  162. with a story that our team put out just this morning so today in the central
  163. this morning so today in the central
  164. this morning so today in the central valley here in california lisa pick off
  165. valley here in california lisa pick off
  166. valley here in california lisa pick off white who's on our big local news team
  167. white who's on our big local news team
  168. white who's on our big local news team teamed up with some public radio
  169. teamed up with some public radio
  170. teamed up with some public radio reporters in the area to do an
  171. reporters in the area to do an
  172. reporters in the area to do an investigation into shootings by the
  173. investigation into shootings by the
  174. investigation into shootings by the bakersfield police department
  175. bakersfield police department
  176. bakersfield police department the local department had been under some
  177. the local department had been under some
  178. the local department had been under some pressure to improve how it handles
  179. pressure to improve how it handles
  180. pressure to improve how it handles interacting with people who have mental
  181. interacting with people who have mental
  182. interacting with people who have mental health issues
  183. health issues
  184. health issues though the department's official
  185. though the department's official
  186. though the department's official position has continued to be in years
  187. position has continued to be in years
  188. position has continued to be in years that that's not really a problem when it
  189. that that's not really a problem when it
  190. that that's not really a problem when it comes to police shootings well what lisa
  191. comes to police shootings well what lisa
  192. comes to police shootings well what lisa and her partners molly and sorith found
  193. and her partners molly and sorith found
  194. and her partners molly and sorith found out did is they built a database where
  195. out did is they built a database where
  196. out did is they built a database where one didn't exist is by filing public
  197. one didn't exist is by filing public
  198. one didn't exist is by filing public records requests that are
  199. records requests that are
  200. records requests that are allow us to access records thanks to a
  201. allow us to access records thanks to a
  202. allow us to access records thanks to a new state law that kind of opened up
  203. new state law that kind of opened up
  204. new state law that kind of opened up what you can get they built a database
  205. what you can get they built a database
  206. what you can get they built a database of every police shooting by the
  207. of every police shooting by the
  208. of every police shooting by the bakersfield police department over a
  209. bakersfield police department over a
  210. bakersfield police department over a five or six year period
  211. five or six year period
  212. five or six year period and then they went through and read them
  213. and then they went through and read them
  214. and then they went through and read them all and so that's a process of like
  215. all and so that's a process of like
  216. all and so that's a process of like gathering big stacks of paper and pdfs
  217. gathering big stacks of paper and pdfs
  218. gathering big stacks of paper and pdfs uh that you get back from your records
  219. uh that you get back from your records
  220. uh that you get back from your records request and then using a sort of
  221. request and then using a sort of
  222. request and then using a sort of database frame of mind to kind of
  223. database frame of mind to kind of
  224. database frame of mind to kind of convert those raw records into a
  225. convert those raw records into a
  226. convert those raw records into a spreadsheet and into something that you
  227. spreadsheet and into something that you
  228. spreadsheet and into something that you can analyze
  229. can analyze
  230. can analyze so that you can say something a little
  231. so that you can say something a little
  232. so that you can say something a little more authoritative than just talk about
  233. more authoritative than just talk about
  234. more authoritative than just talk about one or two cases and if you were to
  235. one or two cases and if you were to
  236. one or two cases and if you were to check out the story which i encourage
  237. check out the story which i encourage
  238. check out the story which i encourage you to do you would see that they found
  239. you to do you would see that they found
  240. you to do you would see that they found that about 40
  241. that about 40
  242. that about 40 of
  243. of
  244. of uses of force during that period were
  245. uses of force during that period were
  246. uses of force during that period were linked to or were against a person who
  247. linked to or were against a person who
  248. linked to or were against a person who who you could say had some sort of
  249. who you could say had some sort of
  250. who you could say had some sort of mental health or sobriety issue at the
  251. mental health or sobriety issue at the
  252. mental health or sobriety issue at the time of the event which is much much
  253. time of the event which is much much
  254. time of the event which is much much larger than the official police estimate
  255. larger than the official police estimate
  256. larger than the official police estimate which was only about three percent so
  257. which was only about three percent so
  258. which was only about three percent so the story unpacks all those issues and
  259. the story unpacks all those issues and
  260. the story unpacks all those issues and talks about how the police department is
  261. talks about how the police department is
  262. talks about how the police department is facing the challenge of really stepping
  263. facing the challenge of really stepping
  264. facing the challenge of really stepping up to take that on and to me this is one
  265. up to take that on and to me this is one
  266. up to take that on and to me this is one one example among many of the type of
  267. one example among many of the type of
  268. one example among many of the type of work that our team tries to do
  269. work that our team tries to do
  270. work that our team tries to do to kind of convert public records and
  271. to kind of convert public records and
  272. to kind of convert public records and sort of the raw information that's out
  273. sort of the raw information that's out
  274. sort of the raw information that's out there in the world and on the internet
  275. there in the world and on the internet
  276. there in the world and on the internet into structured information that we can
  277. into structured information that we can
  278. into structured information that we can interrogate analyze and report on um our
  279. interrogate analyze and report on um our
  280. interrogate analyze and report on um our project big local news is like a website
  281. project big local news is like a website
  282. project big local news is like a website at big local news.org
  283. at big local news.org
  284. at big local news.org and it's also a github profile where
  285. and it's also a github profile where
  286. and it's also a github profile where we're posting you know dozens of open
  287. we're posting you know dozens of open
  288. we're posting you know dozens of open source code repositories that power kind
  289. source code repositories that power kind
  290. source code repositories that power kind of the work that we do and in some cases
  291. of the work that we do and in some cases
  292. of the work that we do and in some cases it's like lisa it's a little more uh ad
  293. it's like lisa it's a little more uh ad
  294. it's like lisa it's a little more uh ad hoc where you're converting
  295. hoc where you're converting
  296. hoc where you're converting semi-structured data into structured
  297. semi-structured data into structured
  298. semi-structured data into structured data but in other cases
  299. data but in other cases
  300. data but in other cases the process is quite a bit more
  301. the process is quite a bit more
  302. the process is quite a bit more automated in our world of data
  303. automated in our world of data
  304. automated in our world of data journalism a really common challenge is
  305. journalism a really common challenge is
  306. journalism a really common challenge is what is you know i think everyone on
  307. what is you know i think everyone on
  308. what is you know i think everyone on this call probably knows about but it's
  309. this call probably knows about but it's
  310. this call probably knows about but it's called web scraping which is really
  311. called web scraping which is really
  312. called web scraping which is really writing a sort of computer script or
  313. writing a sort of computer script or
  314. writing a sort of computer script or spider that can walk across the web pull
  315. spider that can walk across the web pull
  316. spider that can walk across the web pull down pages and then extract kind of
  317. down pages and then extract kind of
  318. down pages and then extract kind of useful information out of those pages
  319. useful information out of those pages
  320. useful information out of those pages into some kind of structured form and
  321. into some kind of structured form and
  322. into some kind of structured form and you know obviously in the case of the
  323. you know obviously in the case of the
  324. you know obviously in the case of the google spider
  325. google spider
  326. google spider that's done in service of creating a
  327. that's done in service of creating a
  328. that's done in service of creating a great search engine
  329. great search engine
  330. great search engine in the case of journalists it's often uh
  331. in the case of journalists it's often uh
  332. in the case of journalists it's often uh about getting databases out of the
  333. about getting databases out of the
  334. about getting databases out of the government that they might not otherwise
  335. government that they might not otherwise
  336. government that they might not otherwise want to get give you or it might be hard
  337. want to get give you or it might be hard
  338. want to get give you or it might be hard to get at
  339. to get at
  340. to get at and so
  341. and so
  342. and so a lot of our projects on our team are
  343. a lot of our projects on our team are
  344. a lot of our projects on our team are going to try to scale up large scale
  345. going to try to scale up large scale
  346. going to try to scale up large scale scrapers that we think will be useful to
  347. scrapers that we think will be useful to
  348. scrapers that we think will be useful to many many news organizations across the
  349. many many news organizations across the
  350. many many news organizations across the country through the power of scale but
  351. country through the power of scale but
  352. country through the power of scale but that the individual news organizations
  353. that the individual news organizations
  354. that the individual news organizations really struggle to pull off on their own
  355. really struggle to pull off on their own
  356. really struggle to pull off on their own and so if you check our github profile
  357. and so if you check our github profile
  358. and so if you check our github profile at big local news you'll see quite a few
  359. at big local news you'll see quite a few
  360. at big local news you'll see quite a few examples of those one that i've been
  361. examples of those one that i've been
  362. examples of those one that i've been working on on recent weeks that doesn't
  363. working on on recent weeks that doesn't
  364. working on on recent weeks that doesn't involve prefect but is sort of more a
  365. involve prefect but is sort of more a
  366. involve prefect but is sort of more a finished product is related to scraping
  367. finished product is related to scraping
  368. finished product is related to scraping layoff notices under something called
  369. layoff notices under something called
  370. layoff notices under something called the warn act and so the warn act is a
  371. the warn act and so the warn act is a
  372. the warn act and so the warn act is a federal law that requires companies to
  373. federal law that requires companies to
  374. federal law that requires companies to to disclose when they lay off a lot of
  375. to disclose when they lay off a lot of
  376. to disclose when they lay off a lot of people at their factory or company
  377. people at their factory or company
  378. people at their factory or company but that data is not consolidated into a
  379. but that data is not consolidated into a
  380. but that data is not consolidated into a single database anywhere each state
  381. single database anywhere each state
  382. single database anywhere each state collects it separately via different
  383. collects it separately via different
  384. collects it separately via different means and so with stanford graduate
  385. means and so with stanford graduate
  386. means and so with stanford graduate students and partners in journalism
  387. students and partners in journalism
  388. students and partners in journalism we've written web scrapers from about 40
  389. we've written web scrapers from about 40
  390. we've written web scrapers from about 40 state websites that flow those all into
  391. state websites that flow those all into
  392. state websites that flow those all into a single database and then allow us to
  393. a single database and then allow us to
  394. a single database and then allow us to write say a slack bot that will notify
  395. write say a slack bot that will notify
  396. write say a slack bot that will notify reporters when a new listing has come in
  397. reporters when a new listing has come in
  398. reporters when a new listing has come in so that's an example of like a scraper
  399. so that's an example of like a scraper
  400. so that's an example of like a scraper in motion just if you're interested in
  401. in motion just if you're interested in
  402. in motion just if you're interested in other big local news projects just
  403. other big local news projects just
  404. other big local news projects just yesterday on our blog at
  405. yesterday on our blog at
  406. yesterday on our blog at biglobalnews.org posts we put up this
  407. biglobalnews.org posts we put up this
  408. biglobalnews.org posts we put up this story by dilsey mercedes mercedes which
  409. story by dilsey mercedes mercedes which
  410. story by dilsey mercedes mercedes which rounds up
  411. rounds up
  412. rounds up um about a dozen different stories from
  413. um about a dozen different stories from
  414. um about a dozen different stories from around the world that our team worked on
  415. around the world that our team worked on
  416. around the world that our team worked on partnering with news organizations to
  417. partnering with news organizations to
  418. partnering with news organizations to turn data into news about covid and so
  419. turn data into news about covid and so
  420. turn data into news about covid and so that includes
  421. that includes
  422. that includes helping journalists in peru build a
  423. helping journalists in peru build a
  424. helping journalists in peru build a database of covet among indigenous
  425. database of covet among indigenous
  426. database of covet among indigenous people in the amazon who weren't being
  427. people in the amazon who weren't being
  428. people in the amazon who weren't being well tracked by the government there it
  429. well tracked by the government there it
  430. well tracked by the government there it also includes working with
  431. also includes working with
  432. also includes working with people in the bay area in california to
  433. people in the bay area in california to
  434. people in the bay area in california to analyze the billions of dollars of
  435. analyze the billions of dollars of
  436. analyze the billions of dollars of emergency loans sent out by the u.s
  437. emergency loans sent out by the u.s
  438. emergency loans sent out by the u.s government under the paycheck protection
  439. government under the paycheck protection
  440. government under the paycheck protection program
  441. program
  442. program and which found pretty wide disparities
  443. and which found pretty wide disparities
  444. and which found pretty wide disparities in where those went and lots and lots of
  445. in where those went and lots and lots of
  446. in where those went and lots and lots of other examples here for different news
  447. other examples here for different news
  448. other examples here for different news organizations across the country and
  449. organizations across the country and
  450. organizations across the country and kind of making those stories happen
  451. kind of making those stories happen
  452. kind of making those stories happen through um software that we write
  453. through um software that we write
  454. through um software that we write through records requests that we file
  455. through records requests that we file
  456. through records requests that we file through partnerships with news orgs and
  457. through partnerships with news orgs and
  458. through partnerships with news orgs and through like tools we put together for
  459. through like tools we put together for
  460. through like tools we put together for reporters kind of the mission of big
  461. reporters kind of the mission of big
  462. reporters kind of the mission of big local news is to make a lot of these
  463. local news is to make a lot of these
  464. local news is to make a lot of these stories happen that wouldn't otherwise
  465. stories happen that wouldn't otherwise
  466. stories happen that wouldn't otherwise you know be stories that are told so
  467. you know be stories that are told so
  468. you know be stories that are told so that's kind of the broad overview of
  469. that's kind of the broad overview of
  470. that's kind of the broad overview of what we do if anybody's interested in
  471. what we do if anybody's interested in
  472. what we do if anybody's interested in learning more about it i would say check
  473. learning more about it i would say check
  474. learning more about it i would say check out the website big local news dot org
  475. out the website big local news dot org
  476. out the website big local news dot org or github.com
  477. or github.com
  478. or github.com big local news
  479. big local news
  480. big local news awesome i think purpose why you guys are
  481. awesome i think purpose why you guys are
  482. awesome i think purpose why you guys are doing this i think but also
  483. doing this i think but also
  484. doing this i think but also one of the reasons why you're talking
  485. one of the reasons why you're talking
  486. one of the reasons why you're talking about it a lot of overlap with potential
  487. about it a lot of overlap with potential
  488. about it a lot of overlap with potential use cases for some of our communities so
  489. use cases for some of our communities so
  490. use cases for some of our communities so i think really great to see the work
  491. i think really great to see the work
  492. i think really great to see the work that you guys are doing and also
  493. that you guys are doing and also
  494. that you guys are doing and also interested i think it's just really
  495. interested i think it's just really
  496. interested i think it's just really interesting a lot of people are doing
  497. interesting a lot of people are doing
  498. interesting a lot of people are doing screen scraping believe it or not um all
  499. screen scraping believe it or not um all
  500. screen scraping believe it or not um all throughout our community and we've seen
  501. throughout our community and we've seen
  502. throughout our community and we've seen it screen scraping and pdf scraping we
  503. it screen scraping and pdf scraping we
  504. it screen scraping and pdf scraping we just had a blog that came out actually
  505. just had a blog that came out actually
  506. just had a blog that came out actually on a pdf called pdf to snowflake by john
  507. on a pdf called pdf to snowflake by john
  508. on a pdf called pdf to snowflake by john luciano on our team which did something
  509. luciano on our team which did something
  510. luciano on our team which did something very very similar
  511. very very similar
  512. very very similar that's awesome i mean parsing pdfs is
  513. that's awesome i mean parsing pdfs is
  514. that's awesome i mean parsing pdfs is really a big challenge for us in the
  515. really a big challenge for us in the
  516. really a big challenge for us in the police accountability work you know you
  517. police accountability work you know you
  518. police accountability work you know you file a records request
  519. file a records request
  520. file a records request and you often get back hundreds or
  521. and you often get back hundreds or
  522. and you often get back hundreds or thousands of pages of pdfs that are in
  523. thousands of pages of pdfs that are in
  524. thousands of pages of pdfs that are in really unusual shapes and sizes you also
  525. really unusual shapes and sizes you also
  526. really unusual shapes and sizes you also we're also getting a lot of unstructured
  527. we're also getting a lot of unstructured
  528. we're also getting a lot of unstructured video as well from those requests like
  529. video as well from those requests like
  530. video as well from those requests like uh
  531. uh
  532. uh body cam footage and so one of our major
  533. body cam footage and so one of our major
  534. body cam footage and so one of our major projects over the next year i won't get
  535. projects over the next year i won't get
  536. projects over the next year i won't get too in-depth about here today but we're
  537. too in-depth about here today but we're
  538. too in-depth about here today but we're definitely looking for tech people to
  539. definitely looking for tech people to
  540. definitely looking for tech people to help on
  541. help on
  542. help on is how can we
  543. is how can we
  544. is how can we build kind of a big database of pdfs and
  545. build kind of a big database of pdfs and
  546. build kind of a big database of pdfs and video
  547. video
  548. video and get some entity extraction going on
  549. and get some entity extraction going on
  550. and get some entity extraction going on so that we can search and find stories
  551. so that we can search and find stories
  552. so that we can search and find stories in that haystack of information
  553. in that haystack of information
  554. in that haystack of information without it costing a ton of money right
  555. without it costing a ton of money right
  556. without it costing a ton of money right and in a way that ultimately yields kind
  557. and in a way that ultimately yields kind
  558. and in a way that ultimately yields kind of stories
  559. of stories
  560. of stories um so where's prefect come in though
  561. um so where's prefect come in though
  562. um so where's prefect come in though right should we get to get to that yeah
  563. right should we get to get to that yeah
  564. right should we get to get to that yeah let's do it okay cool so like one of the
  565. let's do it okay cool so like one of the
  566. let's do it okay cool so like one of the big projects i worked on the last couple
  567. big projects i worked on the last couple
  568. big projects i worked on the last couple years was obviously covering the
  569. years was obviously covering the
  570. years was obviously covering the coronavirus
  571. coronavirus
  572. coronavirus at the los angeles times i was until
  573. at the los angeles times i was until
  574. at the los angeles times i was until recently the editor of the data and
  575. recently the editor of the data and
  576. recently the editor of the data and graphics department and one of our major
  577. graphics department and one of our major
  578. graphics department and one of our major initiatives was to track coronavirus in
  579. initiatives was to track coronavirus in
  580. initiatives was to track coronavirus in the state of california he was starting
  581. the state of california he was starting
  582. the state of california he was starting um in march 2020 we began writing web
  583. um in march 2020 we began writing web
  584. um in march 2020 we began writing web scrapers for all kinds of uh state
  585. scrapers for all kinds of uh state
  586. scrapers for all kinds of uh state websites but also for the 58 counties
  587. websites but also for the 58 counties
  588. websites but also for the 58 counties across the state to try to get as much
  589. across the state to try to get as much
  590. across the state to try to get as much information about covet as possible and
  591. information about covet as possible and
  592. information about covet as possible and we ultimately were able to create a web
  593. we ultimately were able to create a web
  594. we ultimately were able to create a web scraping system that runs open source on
  595. scraping system that runs open source on
  596. scraping system that runs open source on github at california coronavirus
  597. github at california coronavirus
  598. github at california coronavirus scrapers that runs something like 50 to
  599. scrapers that runs something like 50 to
  600. scrapers that runs something like 50 to 100 web scrapers you know all throughout
  601. 100 web scrapers you know all throughout
  602. 100 web scrapers you know all throughout the day and night to pull down data that
  603. the day and night to pull down data that
  604. the day and night to pull down data that then goes on to um
  605. then goes on to um
  606. then goes on to um the
  607. the
  608. the the
  609. the
  610. the you know the the data portal that is
  611. you know the the data portal that is
  612. you know the the data portal that is kind of the most popular page on the
  613. kind of the most popular page on the
  614. kind of the most popular page on the last website where the numbers are then
  615. last website where the numbers are then
  616. last website where the numbers are then surfaced for the general public and in
  617. surfaced for the general public and in
  618. surfaced for the general public and in this case we were able to get by just
  619. this case we were able to get by just
  620. this case we were able to get by just using kind of github actions and little
  621. using kind of github actions and little
  622. using kind of github actions and little itty bitty python scripts because none
  623. itty bitty python scripts because none
  624. itty bitty python scripts because none of them were especially that complicated
  625. of them were especially that complicated
  626. of them were especially that complicated and the data that we were gathering was
  627. and the data that we were gathering was
  628. and the data that we were gathering was pretty
  629. pretty
  630. pretty um you know by silicon valley standards
  631. um you know by silicon valley standards
  632. um you know by silicon valley standards pretty small or simple right a few
  633. pretty small or simple right a few
  634. pretty small or simple right a few hundred or a few thousand records a
  635. hundred or a few thousand records a
  636. hundred or a few thousand records a single file and it really was just
  637. single file and it really was just
  638. single file and it really was just writing about a lot of little scrapers
  639. writing about a lot of little scrapers
  640. writing about a lot of little scrapers doing that we were able to pull it off
  641. doing that we were able to pull it off
  642. doing that we were able to pull it off entirely for free using this kind of
  643. entirely for free using this kind of
  644. entirely for free using this kind of system but that's not you know there's
  645. system but that's not you know there's
  646. system but that's not you know there's often challenges that are bigger than
  647. often challenges that are bigger than
  648. often challenges that are bigger than that where we need bigger and better
  649. that where we need bigger and better
  650. that where we need bigger and better tools which is where prefix comes in and
  651. tools which is where prefix comes in and
  652. tools which is where prefix comes in and you know part of the goal of big local
  653. you know part of the goal of big local
  654. you know part of the goal of big local news is to do scrapers that are bigger
  655. news is to do scrapers that are bigger
  656. news is to do scrapers that are bigger than news organizations can really pull
  657. than news organizations can really pull
  658. than news organizations can really pull off on their own right and there's a
  659. off on their own right and there's a
  660. off on their own right and there's a couple classic examples of this
  661. couple classic examples of this
  662. couple classic examples of this that news organizations across the
  663. that news organizations across the
  664. that news organizations across the country have all kind of tried and
  665. country have all kind of tried and
  666. country have all kind of tried and mostly failed to come up with a
  667. mostly failed to come up with a
  668. mostly failed to come up with a sustainable solution
  669. sustainable solution
  670. sustainable solution and that's really to monitor your local
  671. and that's really to monitor your local
  672. and that's really to monitor your local government and analyze it better through
  673. government and analyze it better through
  674. government and analyze it better through gathering data one would be you know
  675. gathering data one would be you know
  676. gathering data one would be you know your city council everyone has a county
  677. your city council everyone has a county
  678. your city council everyone has a county government or a city government that has
  679. government or a city government that has
  680. government or a city government that has some kind
  681. some kind
  682. some kind of website that is just churning through
  683. of website that is just churning through
  684. of website that is just churning through documents about agendas meetings votes
  685. documents about agendas meetings votes
  686. documents about agendas meetings votes decisions happening in local government
  687. decisions happening in local government
  688. decisions happening in local government and the other is your county courthouse
  689. and the other is your county courthouse
  690. and the other is your county courthouse which is where local judges are making
  691. which is where local judges are making
  692. which is where local judges are making decisions about who goes to jail who
  693. decisions about who goes to jail who
  694. decisions about who goes to jail who doesn't juries as well
  695. doesn't juries as well
  696. doesn't juries as well and a lot of power is being exercised
  697. and a lot of power is being exercised
  698. and a lot of power is being exercised and in pretty much every newsroom in the
  699. and in pretty much every newsroom in the
  700. and in pretty much every newsroom in the united states of a certain size at one
  701. united states of a certain size at one
  702. united states of a certain size at one time or another your local uh newsroom
  703. time or another your local uh newsroom
  704. time or another your local uh newsroom has had a nerd who's tried to write the
  705. has had a nerd who's tried to write the
  706. has had a nerd who's tried to write the scraper for that system
  707. scraper for that system
  708. scraper for that system and and many and it's some fraction of
  709. and and many and it's some fraction of
  710. and and many and it's some fraction of those cases they've actually succeeded
  711. those cases they've actually succeeded
  712. those cases they've actually succeeded hooray right and maybe you came up with
  713. hooray right and maybe you came up with
  714. hooray right and maybe you came up with like a little private search that ran
  715. like a little private search that ran
  716. like a little private search that ran under their desk at the office right or
  717. under their desk at the office right or
  718. under their desk at the office right or maybe you did like a couple good
  719. maybe you did like a couple good
  720. maybe you did like a couple good investigative stories where you analyzed
  721. investigative stories where you analyzed
  722. investigative stories where you analyzed evictions by landlords or
  723. evictions by landlords or
  724. evictions by landlords or racial disparities and sentencing or who
  725. racial disparities and sentencing or who
  726. racial disparities and sentencing or who knows what
  727. knows what
  728. knows what um but you know ultimately that nerd got
  729. um but you know ultimately that nerd got
  730. um but you know ultimately that nerd got another job
  731. another job
  732. another job or
  733. or
  734. or the county website changed and just like
  735. the county website changed and just like
  736. the county website changed and just like keeping those kinds of systems up and
  737. keeping those kinds of systems up and
  738. keeping those kinds of systems up and sustainable there just really hasn't
  739. sustainable there just really hasn't
  740. sustainable there just really hasn't been a solution for that and so that's
  741. been a solution for that and so that's
  742. been a solution for that and so that's that's a like space to use the tech word
  743. that's a like space to use the tech word
  744. that's a like space to use the tech word where big local news is looking to kind
  745. where big local news is looking to kind
  746. where big local news is looking to kind of enter and see if we can we can come
  747. of enter and see if we can we can come
  748. of enter and see if we can we can come up with something that's a little
  749. up with something that's a little
  750. up with something that's a little sturdier and so we have two open source
  751. sturdier and so we have two open source
  752. sturdier and so we have two open source repositories
  753. repositories
  754. repositories you can find on the big local news site
  755. you can find on the big local news site
  756. you can find on the big local news site one is called civic scraper and the
  757. one is called civic scraper and the
  758. one is called civic scraper and the other is called uh court scraper let me
  759. other is called uh court scraper let me
  760. other is called uh court scraper let me call that up real quick and they're you
  761. call that up real quick and they're you
  762. call that up real quick and they're you know just free and open python
  763. know just free and open python
  764. know just free and open python libraries like you've seen before that
  765. libraries like you've seen before that
  766. libraries like you've seen before that have kind of class-based systems
  767. have kind of class-based systems
  768. have kind of class-based systems for
  769. for
  770. for um scraping kind of the major
  771. um scraping kind of the major
  772. um scraping kind of the major platforms or vendors that serve hundreds
  773. platforms or vendors that serve hundreds
  774. platforms or vendors that serve hundreds of local governments across the country
  775. of local governments across the country
  776. of local governments across the country so there's really five or six companies
  777. so there's really five or six companies
  778. so there's really five or six companies that create kind of products they sell
  779. that create kind of products they sell
  780. that create kind of products they sell to local governments to do court records
  781. to local governments to do court records
  782. to local governments to do court records or to do city council records and if you
  783. or to do city council records and if you
  784. or to do city council records and if you can write a scraper
  785. can write a scraper
  786. can write a scraper that extracts the data
  787. that extracts the data
  788. that extracts the data from those vendors sites you get a lot
  789. from those vendors sites you get a lot
  790. from those vendors sites you get a lot of bang for your buck you know there's
  791. of bang for your buck you know there's
  792. of bang for your buck you know there's one vendor called civic plus
  793. one vendor called civic plus
  794. one vendor called civic plus uh who you can find inside this
  795. uh who you can find inside this
  796. uh who you can find inside this repository and if you scrape all the
  797. repository and if you scrape all the
  798. repository and if you scrape all the civic plus websites or all the granicus
  799. civic plus websites or all the granicus
  800. civic plus websites or all the granicus websites that's another one you're
  801. websites that's another one you're
  802. websites that's another one you're you're suddenly gathering
  803. you're suddenly gathering
  804. you're suddenly gathering city council records or capable of
  805. city council records or capable of
  806. city council records or capable of gathering city council records from
  807. gathering city council records from
  808. gathering city council records from literally hundreds of governments across
  809. literally hundreds of governments across
  810. literally hundreds of governments across the country and so our goal is to to try
  811. the country and so our goal is to to try
  812. the country and so our goal is to to try to get as much bang for the buck as
  813. to get as much bang for the buck as
  814. to get as much bang for the buck as possible to write scrapers that really
  815. possible to write scrapers that really
  816. possible to write scrapers that really target the large vendors that serve lots
  817. target the large vendors that serve lots
  818. target the large vendors that serve lots of areas and then to run
  819. of areas and then to run
  820. of areas and then to run spiders that then
  821. spiders that then
  822. spiders that then jump across as many local venues as
  823. jump across as many local venues as
  824. jump across as many local venues as possible and try to build one big
  825. possible and try to build one big
  826. possible and try to build one big national database of
  827. national database of
  828. national database of city council agendas and county court
  829. city council agendas and county court
  830. city council agendas and county court decisions
  831. decisions
  832. decisions and so
  833. and so
  834. and so we're at a stage of these projects where
  835. we're at a stage of these projects where
  836. we're at a stage of these projects where we've got kind of the draft python
  837. we've got kind of the draft python
  838. we've got kind of the draft python applications we've got some of these
  839. applications we've got some of these
  840. applications we've got some of these scrapers ready to go but now we actually
  841. scrapers ready to go but now we actually
  842. scrapers ready to go but now we actually need to run the damn things right and
  843. need to run the damn things right and
  844. need to run the damn things right and github actions is great but if you're
  845. github actions is great but if you're
  846. github actions is great but if you're going to scrape
  847. going to scrape
  848. going to scrape 500 county court websites or whatever it
  849. 500 county court websites or whatever it
  850. 500 county court websites or whatever it ends up being and each one of those
  851. ends up being and each one of those
  852. ends up being and each one of those scrapers is going to take a really
  853. scrapers is going to take a really
  854. scrapers is going to take a really extended amount of time
  855. extended amount of time
  856. extended amount of time and if they're going to need to store
  857. and if they're going to need to store
  858. and if they're going to need to store data a lot of data in a pretty big data
  859. data a lot of data in a pretty big data
  860. data a lot of data in a pretty big data pool somewhere you know then that kind
  861. pool somewhere you know then that kind
  862. pool somewhere you know then that kind of free tool you're using on github
  863. of free tool you're using on github
  864. of free tool you're using on github actions doesn't look so great anymore
  865. actions doesn't look so great anymore
  866. actions doesn't look so great anymore right and you really need to find a way
  867. right and you really need to find a way
  868. right and you really need to find a way to scale up and i'll be honest with you
  869. to scale up and i'll be honest with you
  870. to scale up and i'll be honest with you within the sort of like hacker world of
  871. within the sort of like hacker world of
  872. within the sort of like hacker world of journalism where there isn't a lot of
  873. journalism where there isn't a lot of
  874. journalism where there isn't a lot of money to be made
  875. money to be made
  876. money to be made and we're all kind of like people like
  877. and we're all kind of like people like
  878. and we're all kind of like people like me people who went to journalism school
  879. me people who went to journalism school
  880. me people who went to journalism school and are like faker coders
  881. and are like faker coders
  882. and are like faker coders you know there really hasn't this
  883. you know there really hasn't this
  884. you know there really hasn't this problem like really hasn't been solved
  885. problem like really hasn't been solved
  886. problem like really hasn't been solved within our little like niche and so
  887. within our little like niche and so
  888. within our little like niche and so this is part of where we're getting
  889. this is part of where we're getting
  890. this is part of where we're getting stanford behind it we're trying out a
  891. stanford behind it we're trying out a
  892. stanford behind it we're trying out a tool like prefect our goal is to kind of
  893. tool like prefect our goal is to kind of
  894. tool like prefect our goal is to kind of go a little further than you than the
  895. go a little further than you than the
  896. go a little further than you than the news organizations have really gone
  897. news organizations have really gone
  898. news organizations have really gone before i suspect some of these problems
  899. before i suspect some of these problems
  900. before i suspect some of these problems might seem small to people who work at
  901. might seem small to people who work at
  902. might seem small to people who work at big tech but they're
  903. big tech but they're
  904. big tech but they're you know we're trying to get over a
  905. you know we're trying to get over a
  906. you know we're trying to get over a pretty big hump for journalism to try to
  907. pretty big hump for journalism to try to
  908. pretty big hump for journalism to try to step it up to another level and so
  909. step it up to another level and so
  910. step it up to another level and so what i've been working on the last
  911. what i've been working on the last
  912. what i've been working on the last couple months isn't really these
  913. couple months isn't really these
  914. couple months isn't really these scrapers it's the prefect integration
  915. scrapers it's the prefect integration
  916. scrapers it's the prefect integration and so
  917. and so
  918. and so um you can see here here's here this is
  919. um you can see here here's here this is
  920. um you can see here here's here this is the private repository now and so this
  921. the private repository now and so this
  922. the private repository now and so this is a private repository called the civic
  923. is a private repository called the civic
  924. is a private repository called the civic prefect flow
  925. prefect flow
  926. prefect flow and this is my kind of current effort to
  927. and this is my kind of current effort to
  928. and this is my kind of current effort to take that open source civic scraper code
  929. take that open source civic scraper code
  930. take that open source civic scraper code we were just looking at from pi pi and
  931. we were just looking at from pi pi and
  932. we were just looking at from pi pi and the open source world of python and
  933. the open source world of python and
  934. the open source world of python and install it so that it's ready to be
  935. install it so that it's ready to be
  936. install it so that it's ready to be plugged into a kubernetes cluster
  937. plugged into a kubernetes cluster
  938. plugged into a kubernetes cluster running in google cloud
  939. running in google cloud
  940. running in google cloud a sort of whole prefect kit so that
  941. a sort of whole prefect kit so that
  942. a sort of whole prefect kit so that we'll be able to schedule tasks that
  943. we'll be able to schedule tasks that
  944. we'll be able to schedule tasks that will run across hundreds or maybe even
  945. will run across hundreds or maybe even
  946. will run across hundreds or maybe even thousands of sites before we're done and
  947. thousands of sites before we're done and
  948. thousands of sites before we're done and so
  949. so
  950. so this is um
  951. this is um
  952. this is um this is you know pretty basic prefix
  953. this is you know pretty basic prefix
  954. this is you know pretty basic prefix stuff but you can see that it's based on
  955. stuff but you can see that it's based on
  956. stuff but you can see that it's based on a github template which is also open
  957. a github template which is also open
  958. a github template which is also open source so we've created an open source
  959. source so we've created an open source
  960. source so we've created an open source template called prefect flow template
  961. template called prefect flow template
  962. template called prefect flow template which is kind of my the current like
  963. which is kind of my the current like
  964. which is kind of my the current like best knowledge that i've been able to
  965. best knowledge that i've been able to
  966. best knowledge that i've been able to assemble for myself personally by
  967. assemble for myself personally by
  968. assemble for myself personally by bugging chris and his colleagues at
  969. bugging chris and his colleagues at
  970. bugging chris and his colleagues at slack by doing a lot of hacking
  971. slack by doing a lot of hacking
  972. slack by doing a lot of hacking uh on my own time at work is this is
  973. uh on my own time at work is this is
  974. uh on my own time at work is this is kind of the bare bones for how you would
  975. kind of the bare bones for how you would
  976. kind of the bare bones for how you would take a prefect flow and plug it into
  977. take a prefect flow and plug it into
  978. take a prefect flow and plug it into kubernetes and um
  979. kubernetes and um
  980. kubernetes and um and google google cloud and so you can
  981. and google google cloud and so you can
  982. and google google cloud and so you can see it here some pretty basic like hacks
  983. see it here some pretty basic like hacks
  984. see it here some pretty basic like hacks i've come up with for managing
  985. i've come up with for managing
  986. i've come up with for managing development versus production
  987. development versus production
  988. development versus production environments
  989. environments
  990. environments where the code itself is going to be
  991. where the code itself is going to be
  992. where the code itself is going to be stuffed up into docker inside the google
  993. stuffed up into docker inside the google
  994. stuffed up into docker inside the google artifact registry
  995. artifact registry
  996. artifact registry and
  997. and
  998. and the uh run is going to be done by a
  999. the uh run is going to be done by a
  1000. the uh run is going to be done by a kubernetes agent the size of which is
  1001. kubernetes agent the size of which is
  1002. kubernetes agent the size of which is kind of dictated by some of this config
  1003. kind of dictated by some of this config
  1004. kind of dictated by some of this config here
  1005. here
  1006. here and this code
  1007. and this code
  1008. and this code is pretty much ready to go for anyone
  1009. is pretty much ready to go for anyone
  1010. is pretty much ready to go for anyone who would want to experiment with it or
  1011. who would want to experiment with it or
  1012. who would want to experiment with it or try to use it
  1013. try to use it
  1014. try to use it besides just kind of figuring out how to
  1015. besides just kind of figuring out how to
  1016. besides just kind of figuring out how to manage the toggle between the two
  1017. manage the toggle between the two
  1018. manage the toggle between the two environments i'm pretty proud of what
  1019. environments i'm pretty proud of what
  1020. environments i'm pretty proud of what we've been able to do here with uh
  1021. we've been able to do here with uh
  1022. we've been able to do here with uh github actions integration where we have
  1023. github actions integration where we have
  1024. github actions integration where we have a workflow that's included with this
  1025. a workflow that's included with this
  1026. a workflow that's included with this that has kind of a continuous deployment
  1027. that has kind of a continuous deployment
  1028. that has kind of a continuous deployment so that every time you you've ever done
  1029. so that every time you you've ever done
  1030. so that every time you you've ever done a github release every time you do a
  1031. a github release every time you do a
  1032. a github release every time you do a release on github or you kick out a new
  1033. release on github or you kick out a new
  1034. release on github or you kick out a new version this will actually trigger a
  1035. version this will actually trigger a
  1036. version this will actually trigger a deployment into google cloud
  1037. deployment into google cloud
  1038. deployment into google cloud um automatically
  1039. um automatically
  1040. um automatically where um
  1041. where um
  1042. where um the docker in image with the code that's
  1043. the docker in image with the code that's
  1044. the docker in image with the code that's going to be run will be re-uploaded and
  1045. going to be run will be re-uploaded and
  1046. going to be run will be re-uploaded and versioned
  1047. versioned
  1048. versioned into google artifact registry and the
  1049. into google artifact registry and the
  1050. into google artifact registry and the prefect flow will be re-registered with
  1051. prefect flow will be re-registered with
  1052. prefect flow will be re-registered with our prefect account kind of
  1053. our prefect account kind of
  1054. our prefect account kind of automatically and so just through a
  1055. automatically and so just through a
  1056. automatically and so just through a little bit of kind of configuration of
  1057. little bit of kind of configuration of
  1058. little bit of kind of configuration of some of these goofy tools you can make
  1059. some of these goofy tools you can make
  1060. some of these goofy tools you can make it so that you're able to push out
  1061. it so that you're able to push out
  1062. it so that you're able to push out new iterations of
  1063. new iterations of
  1064. new iterations of the kind of flow um without it being a
  1065. the kind of flow um without it being a
  1066. the kind of flow um without it being a big
  1067. big
  1068. big goddamn hassle to like
  1069. goddamn hassle to like
  1070. goddamn hassle to like shove shove the vcr tape into the google
  1071. shove shove the vcr tape into the google
  1072. shove shove the vcr tape into the google um you know vcr
  1073. um you know vcr
  1074. um you know vcr and so that all kind of ends up inside a
  1075. and so that all kind of ends up inside a
  1076. and so that all kind of ends up inside a kubernetes cluster which you can see
  1077. kubernetes cluster which you can see
  1078. kubernetes cluster which you can see here where right now the only thing
  1079. here where right now the only thing
  1080. here where right now the only thing running is currently our prefect agent
  1081. running is currently our prefect agent
  1082. running is currently our prefect agent which is sort of you know the all was on
  1083. which is sort of you know the all was on
  1084. which is sort of you know the all was on watcher that's waiting to be told to do
  1085. watcher that's waiting to be told to do
  1086. watcher that's waiting to be told to do something
  1087. something
  1088. something and uh that's configured through a
  1089. and uh that's configured through a
  1090. and uh that's configured through a separate private repository by the way
  1091. separate private repository by the way
  1092. separate private repository by the way which if people are interested in how to
  1093. which if people are interested in how to
  1094. which if people are interested in how to do a prefect agent which is a little
  1095. do a prefect agent which is a little
  1096. do a prefect agent which is a little different from doing a prefect flow and
  1097. different from doing a prefect flow and
  1098. different from doing a prefect flow and it's kind of part of learning how to how
  1099. it's kind of part of learning how to how
  1100. it's kind of part of learning how to how to uh learning how to dougie within this
  1101. to uh learning how to dougie within this
  1102. to uh learning how to dougie within this kind of system we've got a private um
  1103. kind of system we've got a private um
  1104. kind of system we've got a private um repo there i'd be happy to share some of
  1105. repo there i'd be happy to share some of
  1106. repo there i'd be happy to share some of the stuff i've learned on and we
  1107. the stuff i've learned on and we
  1108. the stuff i've learned on and we actually did a post on our blog
  1109. actually did a post on our blog
  1110. actually did a post on our blog which walks through all the steps on how
  1111. which walks through all the steps on how
  1112. which walks through all the steps on how to deploy your prefect agent to google
  1113. to deploy your prefect agent to google
  1114. to deploy your prefect agent to google kubernetes engine
  1115. kubernetes engine
  1116. kubernetes engine um which is just a lot of really
  1117. um which is just a lot of really
  1118. um which is just a lot of really annoying g cloud commands um which i'd
  1119. annoying g cloud commands um which i'd
  1120. annoying g cloud commands um which i'd be happy to talk about but i don't want
  1121. be happy to talk about but i don't want
  1122. be happy to talk about but i don't want to bore you but the result is is that
  1123. to bore you but the result is is that
  1124. to bore you but the result is is that when the prefix scheduled task is
  1125. when the prefix scheduled task is
  1126. when the prefix scheduled task is kickstarted within prefect it's able the
  1127. kickstarted within prefect it's able the
  1128. kickstarted within prefect it's able the agent will pick it up and then we'll
  1129. agent will pick it up and then we'll
  1130. agent will pick it up and then we'll just create workers here
  1131. just create workers here
  1132. just create workers here as dictated in the flow
  1133. as dictated in the flow
  1134. as dictated in the flow uh and they'll do the thing and keep
  1135. uh and they'll do the thing and keep
  1136. uh and they'll do the thing and keep moving and so
  1137. moving and so
  1138. moving and so we're at a stage now where we've kind of
  1139. we're at a stage now where we've kind of
  1140. we're at a stage now where we've kind of like learned enough prefect to get all
  1141. like learned enough prefect to get all
  1142. like learned enough prefect to get all the kind of ducks in a row and the
  1143. the kind of ducks in a row and the
  1144. the kind of ducks in a row and the pieces in place and over the next few
  1145. pieces in place and over the next few
  1146. pieces in place and over the next few weeks we're going to look to really
  1147. weeks we're going to look to really
  1148. weeks we're going to look to really start slotting in the scraper and
  1149. start slotting in the scraper and
  1150. start slotting in the scraper and scaling up how much data we're gathering
  1151. scaling up how much data we're gathering
  1152. scaling up how much data we're gathering that's something that's that's still
  1153. that's something that's that's still
  1154. that's something that's that's still actually ahead of us but my hope is is
  1155. actually ahead of us but my hope is is
  1156. actually ahead of us but my hope is is that by the end of this month we'll
  1157. that by the end of this month we'll
  1158. that by the end of this month we'll really be skate we'll be running one of
  1159. really be skate we'll be running one of
  1160. really be skate we'll be running one of the biggest scrapers in journalism i
  1161. the biggest scrapers in journalism i
  1162. the biggest scrapers in journalism i think by the end of the month which is
  1163. think by the end of the month which is
  1164. think by the end of the month which is pretty exciting then we got to figure
  1165. pretty exciting then we got to figure
  1166. pretty exciting then we got to figure out what we're going to do with all that
  1167. out what we're going to do with all that
  1168. out what we're going to do with all that data which is a whole other challenge of
  1169. data which is a whole other challenge of
  1170. data which is a whole other challenge of analysis and
  1171. data science but that's a separate kind
  1172. data science but that's a separate kind
  1173. data science but that's a separate kind of question entirely i suppose that's
  1174. of question entirely i suppose that's
  1175. of question entirely i suppose that's awesome it's really funny i think two
  1176. awesome it's really funny i think two
  1177. awesome it's really funny i think two weeks in a row we've had community
  1178. weeks in a row we've had community
  1179. weeks in a row we've had community members on predict live in two weeks in
  1180. members on predict live in two weeks in
  1181. members on predict live in two weeks in a row it's been the same pattern of
  1182. a row it's been the same pattern of
  1183. a row it's been the same pattern of deploying a docker agent via github
  1184. deploying a docker agent via github
  1185. deploying a docker agent via github actions um or so it's it's it's
  1186. actions um or so it's it's it's
  1187. actions um or so it's it's it's interesting to see you're doing it in
  1188. interesting to see you're doing it in
  1189. interesting to see you're doing it in google cloud i think that henning who
  1190. google cloud i think that henning who
  1191. google cloud i think that henning who was on last week was doing a picture but
  1192. was on last week was doing a picture but
  1193. was on last week was doing a picture but it's an interesting pattern that that
  1194. it's an interesting pattern that that
  1195. it's an interesting pattern that that we're seeing out there and i think will
  1196. we're seeing out there and i think will
  1197. we're seeing out there and i think will be really really helpful um if other
  1198. be really really helpful um if other
  1199. be really really helpful um if other people were to see it so i will
  1200. people were to see it so i will
  1201. people were to see it so i will highlight this blog post when we when we
  1202. highlight this blog post when we when we
  1203. highlight this blog post when we when we post it on youtube too
  1204. post it on youtube too
  1205. post it on youtube too the trick is really getting into this
  1206. the trick is really getting into this
  1207. the trick is really getting into this thing called the google artifact
  1208. thing called the google artifact
  1209. thing called the google artifact registry which is really like their
  1210. registry which is really like their
  1211. registry which is really like their private version of pi pi in some ways
  1212. private version of pi pi in some ways
  1213. private version of pi pi in some ways right it's a place where you can like
  1214. right it's a place where you can like
  1215. right it's a place where you can like ship private python packages private
  1216. ship private python packages private
  1217. ship private python packages private docker images
  1218. docker images
  1219. docker images and then um version them so they can
  1220. and then um version them so they can
  1221. and then um version them so they can kind of get kicked out so here's the
  1222. kind of get kicked out so here's the
  1223. kind of get kicked out so here's the docker image for our agent and here's
  1224. docker image for our agent and here's
  1225. docker image for our agent and here's the docker image for that prefect flow
  1226. the docker image for that prefect flow
  1227. the docker image for that prefect flow and every time we do a github release it
  1228. and every time we do a github release it
  1229. and every time we do a github release it just stuffs another version of it kind
  1230. just stuffs another version of it kind
  1231. just stuffs another version of it kind of in here right
  1232. of in here right
  1233. of in here right um so you can see this one's been
  1234. um so you can see this one's been
  1235. um so you can see this one's been versioned a few times and
  1236. versioned a few times and
  1237. versioned a few times and um and so like learning that is kind of
  1238. um and so like learning that is kind of
  1239. um and so like learning that is kind of the trick uh on the google cloud side
  1240. the trick uh on the google cloud side
  1241. the trick uh on the google cloud side and then the github action side we have
  1242. and then the github action side we have
  1243. and then the github action side we have another post that kind of shows this too
  1244. another post that kind of shows this too
  1245. another post that kind of shows this too which is um
  1246. which is um
  1247. which is um how to push tagged release to google
  1248. how to push tagged release to google
  1249. how to push tagged release to google artifact registry with github action
  1250. artifact registry with github action
  1251. artifact registry with github action this really gets at the challenge of
  1252. this really gets at the challenge of
  1253. this really gets at the challenge of integration with github which is not
  1254. integration with github which is not
  1255. integration with github which is not especially hard but it requires you to
  1256. especially hard but it requires you to
  1257. especially hard but it requires you to create you create this thing called a
  1258. create you create this thing called a
  1259. create you create this thing called a service worker or a workload identity
  1260. service worker or a workload identity
  1261. service worker or a workload identity federation which is really just a bunch
  1262. federation which is really just a bunch
  1263. federation which is really just a bunch of gobbledygook for like how do you
  1264. of gobbledygook for like how do you
  1265. of gobbledygook for like how do you create a special user that github is
  1266. create a special user that github is
  1267. create a special user that github is allowed to use to muck around with your
  1268. allowed to use to muck around with your
  1269. allowed to use to muck around with your google cloud right
  1270. google cloud right
  1271. google cloud right but like just like creating the user for
  1272. but like just like creating the user for
  1273. but like just like creating the user for the action to talk to google cloud
  1274. the action to talk to google cloud
  1275. the action to talk to google cloud requires like six or seven like crazy g
  1276. requires like six or seven like crazy g
  1277. requires like six or seven like crazy g cloud commands and like permissions and
  1278. cloud commands and like permissions and
  1279. cloud commands and like permissions and so the first time you do it it's kind of
  1280. so the first time you do it it's kind of
  1281. so the first time you do it it's kind of wild and crazy but once you once you're
  1282. wild and crazy but once you once you're
  1283. wild and crazy but once you once you're done with it it's pretty simple to just
  1284. done with it it's pretty simple to just
  1285. done with it it's pretty simple to just like plug it in
  1286. like plug it in
  1287. like plug it in you you have a talent for synthesizing
  1288. you you have a talent for synthesizing
  1289. you you have a talent for synthesizing things in a really i think
  1290. things in a really i think
  1291. things in a really i think interesting way and i i would agree with
  1292. interesting way and i i would agree with
  1293. interesting way and i i would agree with you it is just a bunch of gobbledygook
  1294. you it is just a bunch of gobbledygook
  1295. you it is just a bunch of gobbledygook in order to to create a essential oh my
  1296. in order to to create a essential oh my
  1297. in order to to create a essential oh my god a workload identity pool versus like
  1298. god a workload identity pool versus like
  1299. god a workload identity pool versus like a workload identity provider like you
  1300. a workload identity provider like you
  1301. a workload identity provider like you know come on yeah
  1302. know come on yeah
  1303. know come on yeah um so we you and i were talking before
  1304. um so we you and i were talking before
  1305. um so we you and i were talking before we went live a little bit about your
  1306. we went live a little bit about your
  1307. we went live a little bit about your experience we started started with
  1308. experience we started started with
  1309. experience we started started with prefect 2 and i think that a lot of a
  1310. prefect 2 and i think that a lot of a
  1311. prefect 2 and i think that a lot of a lot of users
  1312. lot of users
  1313. lot of users experienced maybe some of the same
  1314. experienced maybe some of the same
  1315. experienced maybe some of the same challenges you may have if you i'd love
  1316. challenges you may have if you i'd love
  1317. challenges you may have if you i'd love for you to talk about those a little bit
  1318. for you to talk about those a little bit
  1319. for you to talk about those a little bit briefly and then maybe we can help to
  1320. briefly and then maybe we can help to
  1321. briefly and then maybe we can help to push you yeah in the right direction and
  1322. push you yeah in the right direction and
  1323. push you yeah in the right direction and then secondly if you have any any
  1324. then secondly if you have any any
  1325. then secondly if you have any any thoughts or experience on prefect 2
  1326. thoughts or experience on prefect 2
  1327. thoughts or experience on prefect 2 which is or orion yeah i mean so i'm a
  1328. which is or orion yeah i mean so i'm a
  1329. which is or orion yeah i mean so i'm a python guy i've been doing python for 15
  1330. python guy i've been doing python for 15
  1331. python guy i've been doing python for 15 years or whatever i wouldn't say i'm an
  1332. years or whatever i wouldn't say i'm an
  1333. years or whatever i wouldn't say i'm an expert i'm more of like a journalist
  1334. expert i'm more of like a journalist
  1335. expert i'm more of like a journalist hacker type but i'm comfortable with
  1336. hacker type but i'm comfortable with
  1337. hacker type but i'm comfortable with python it's what i like
  1338. python it's what i like
  1339. python it's what i like i have opinions about it you know what i
  1340. i have opinions about it you know what i
  1341. i have opinions about it you know what i mean i've gotten to that stage but like
  1342. mean i've gotten to that stage but like
  1343. mean i've gotten to that stage but like um and so prefect i think is a really
  1344. um and so prefect i think is a really
  1345. um and so prefect i think is a really great thing for a python person to get
  1346. great thing for a python person to get
  1347. great thing for a python person to get started because it's really
  1348. started because it's really
  1349. started because it's really well put together python code i think
  1350. well put together python code i think
  1351. well put together python code i think anyone who's a veteran at it will kind
  1352. anyone who's a veteran at it will kind
  1353. anyone who's a veteran at it will kind of recognize that pretty quickly they
  1354. of recognize that pretty quickly they
  1355. of recognize that pretty quickly they have a lot of really good examples of
  1356. have a lot of really good examples of
  1357. have a lot of really good examples of how they made flows happen to me the
  1358. how they made flows happen to me the
  1359. how they made flows happen to me the real challenge wasn't the prefect code
  1360. real challenge wasn't the prefect code
  1361. real challenge wasn't the prefect code or the python which i thought came
  1362. or the python which i thought came
  1363. or the python which i thought came pretty quick to me the challenge was
  1364. pretty quick to me the challenge was
  1365. pretty quick to me the challenge was figuring out how to integrate it with
  1366. figuring out how to integrate it with
  1367. figuring out how to integrate it with google cloud and my hosting provider
  1368. google cloud and my hosting provider
  1369. google cloud and my hosting provider right where it's like okay so i've got
  1370. right where it's like okay so i've got
  1371. right where it's like okay so i've got this flow
  1372. this flow
  1373. this flow like how do i get it up somewhere so
  1374. like how do i get it up somewhere so
  1375. like how do i get it up somewhere so it's like running and like what exactly
  1376. it's like running and like what exactly
  1377. it's like running and like what exactly is an agent as opposed to my flow and
  1378. is an agent as opposed to my flow and
  1379. is an agent as opposed to my flow and how do i make all those kind of things
  1380. how do i make all those kind of things
  1381. how do i make all those kind of things like run within the provider
  1382. like run within the provider
  1383. like run within the provider was really the challenge and the way i
  1384. was really the challenge and the way i
  1385. was really the challenge and the way i got there was just like through you know
  1386. got there was just like through you know
  1387. got there was just like through you know embarrassing questions and kind of
  1388. embarrassing questions and kind of
  1389. embarrassing questions and kind of stumbling my way in the prefect slack so
  1390. stumbling my way in the prefect slack so
  1391. stumbling my way in the prefect slack so i really wouldn't i mean it worked great
  1392. i really wouldn't i mean it worked great
  1393. i really wouldn't i mean it worked great for me i would really say anyone who's
  1394. for me i would really say anyone who's
  1395. for me i would really say anyone who's getting started get into the prefect
  1396. getting started get into the prefect
  1397. getting started get into the prefect slack and just like just ask the
  1398. slack and just like just ask the
  1399. slack and just like just ask the questions like how do i do this
  1400. questions like how do i do this
  1401. questions like how do i do this and you know you might be that you know
  1402. and you know you might be that you know
  1403. and you know you might be that you know it could be that you're the first person
  1404. it could be that you're the first person
  1405. it could be that you're the first person to ask this or that but you probably
  1406. to ask this or that but you probably
  1407. to ask this or that but you probably aren't and the folks in there can kind
  1408. aren't and the folks in there can kind
  1409. aren't and the folks in there can kind of point you to some other resources
  1410. of point you to some other resources
  1411. of point you to some other resources that's a big reason i wrote these blog
  1412. that's a big reason i wrote these blog
  1413. that's a big reason i wrote these blog posts as well myself and put together
  1414. posts as well myself and put together
  1415. posts as well myself and put together the open source flow template is one i
  1416. the open source flow template is one i
  1417. the open source flow template is one i want our team and other people on my
  1418. want our team and other people on my
  1419. want our team and other people on my team to be able to learn this because
  1420. team to be able to learn this because
  1421. team to be able to learn this because i'm like our kind of explorer on this
  1422. i'm like our kind of explorer on this
  1423. i'm like our kind of explorer on this but i also just want to make it so other
  1424. but i also just want to make it so other
  1425. but i also just want to make it so other folks can pick up the tech without it
  1426. folks can pick up the tech without it
  1427. folks can pick up the tech without it being a big hassle too absolutely
  1428. being a big hassle too absolutely
  1429. being a big hassle too absolutely have you you thought have you read about
  1430. have you you thought have you read about
  1431. have you you thought have you read about 2.0 at all or have you thought about i
  1432. 2.0 at all or have you thought about i
  1433. 2.0 at all or have you thought about i know it's like a thing and it's it's
  1434. know it's like a thing and it's it's
  1435. know it's like a thing and it's it's it's pitched to be better but i to be
  1436. it's pitched to be better but i to be
  1437. it's pitched to be better but i to be honest with you i haven't even looked at
  1438. honest with you i haven't even looked at
  1439. honest with you i haven't even looked at it yep yet no worries um and then
  1440. it yep yet no worries um and then
  1441. it yep yet no worries um and then scaling is one thing i wanted to touch
  1442. scaling is one thing i wanted to touch
  1443. scaling is one thing i wanted to touch on too that we did talk about earlier a
  1444. on too that we did talk about earlier a
  1445. on too that we did talk about earlier a little bit i know you probably really
  1446. little bit i know you probably really
  1447. little bit i know you probably really have no like no idea what like what the
  1448. have no like no idea what like what the
  1449. have no like no idea what like what the core usage will be right but um how much
  1450. core usage will be right but um how much
  1451. core usage will be right but um how much how much data are you looking to scrape
  1452. how much data are you looking to scrape
  1453. how much data are you looking to scrape broadly if you can say that maybe number
  1454. broadly if you can say that maybe number
  1455. broadly if you can say that maybe number of pages yeah i mean we haven't really
  1456. of pages yeah i mean we haven't really
  1457. of pages yeah i mean we haven't really done the estimate which we should
  1458. done the estimate which we should
  1459. done the estimate which we should you know um but i think it's gonna be
  1460. you know um but i think it's gonna be
  1461. you know um but i think it's gonna be millions of artifacts you know what i
  1462. millions of artifacts you know what i
  1463. millions of artifacts you know what i mean it's gonna be
  1464. mean it's gonna be
  1465. mean it's gonna be because if you think about it like
  1466. because if you think about it like
  1467. because if you think about it like you know if you go through the major
  1468. you know if you go through the major
  1469. you know if you go through the major courthouses in the state of california
  1470. courthouses in the state of california
  1471. courthouses in the state of california you know the five or ten biggest
  1472. you know the five or ten biggest
  1473. you know the five or ten biggest counties are doing tens of thousands of
  1474. counties are doing tens of thousands of
  1475. counties are doing tens of thousands of cases every year you know in their local
  1476. cases every year you know in their local
  1477. cases every year you know in their local county courthouses and then every one of
  1478. county courthouses and then every one of
  1479. county courthouses and then every one of those cases may have multiple records
  1480. those cases may have multiple records
  1481. those cases may have multiple records depending on how the website's set up
  1482. depending on how the website's set up
  1483. depending on how the website's set up and so i think that stuff's going to
  1484. and so i think that stuff's going to
  1485. and so i think that stuff's going to multiply pretty quickly so i think we'll
  1486. multiply pretty quickly so i think we'll
  1487. multiply pretty quickly so i think we'll probably end up in the millions or tens
  1488. probably end up in the millions or tens
  1489. probably end up in the millions or tens of millions of records
  1490. of millions of records
  1491. of millions of records without much trouble without before we
  1492. without much trouble without before we
  1493. without much trouble without before we get too far
  1494. get too far
  1495. get too far you know um so i think it's it's going
  1496. you know um so i think it's it's going
  1497. you know um so i think it's it's going to be in that ballpark which for
  1498. to be in that ballpark which for
  1499. to be in that ballpark which for journalism is big you know
  1500. journalism is big you know
  1501. journalism is big you know yeah and we i mean we're really
  1502. yeah and we i mean we're really
  1503. yeah and we i mean we're really interested to follow the
  1504. interested to follow the
  1505. interested to follow the failing story and i think it'll be
  1506. failing story and i think it'll be
  1507. failing story and i think it'll be really really cool to i mean you have to
  1508. really really cool to i mean you have to
  1509. really really cool to i mean you have to have you back on here at least to see uh
  1510. have you back on here at least to see uh
  1511. have you back on here at least to see uh how fast how much you've scaled and how
  1512. how fast how much you've scaled and how
  1513. how fast how much you've scaled and how totally yeah and i think because there's
  1514. totally yeah and i think because there's
  1515. totally yeah and i think because there's going to be two things to it there's the
  1516. going to be two things to it there's the
  1517. going to be two things to it there's the workers to just go get it but then
  1518. workers to just go get it but then
  1519. workers to just go get it but then there's just like the static files which
  1520. there's just like the static files which
  1521. there's just like the static files which is like where do we stick them and then
  1522. is like where do we stick them and then
  1523. is like where do we stick them and then how do we do anything with them
  1524. how do we do anything with them
  1525. how do we do anything with them efficiently this is one thing that you
  1526. efficiently this is one thing that you
  1527. efficiently this is one thing that you know you know some of my good friends
  1528. know you know some of my good friends
  1529. know you know some of my good friends work on
  1530. work on
  1531. work on the panama papers or the pandora papers
  1532. the panama papers or the pandora papers
  1533. the panama papers or the pandora papers which are projects i bet people on
  1534. which are projects i bet people on
  1535. which are projects i bet people on you've heard about which are these huge
  1536. you've heard about which are these huge
  1537. you've heard about which are these huge leaks from central american law firms
  1538. leaks from central american law firms
  1539. leaks from central american law firms um
  1540. um
  1541. um that you know that hold offshore
  1542. that you know that hold offshore
  1543. that you know that hold offshore accounts for the world's wealthy and
  1544. accounts for the world's wealthy and
  1545. accounts for the world's wealthy and they have to deal with you know millions
  1546. they have to deal with you know millions
  1547. they have to deal with you know millions of pdfs and records for their stories
  1548. of pdfs and records for their stories
  1549. of pdfs and records for their stories and they find
  1550. and they find
  1551. and they find that um it's so costly to run like a
  1552. that um it's so costly to run like a
  1553. that um it's so costly to run like a data science operation across the entire
  1554. data science operation across the entire
  1555. data science operation across the entire set that they tend not just to do it you
  1556. set that they tend not just to do it you
  1557. set that they tend not just to do it you know there's a way in which i think
  1558. know there's a way in which i think
  1559. know there's a way in which i think there's a lot of potential
  1560. there's a lot of potential
  1561. there's a lot of potential to take what people have you know
  1562. to take what people have you know
  1563. to take what people have you know developed in machine learning and and ai
  1564. developed in machine learning and and ai
  1565. developed in machine learning and and ai techniques to
  1566. techniques to
  1567. techniques to to try to to bring more out of these
  1568. to try to to bring more out of these
  1569. to try to to bring more out of these like data sets that have a high public
  1570. like data sets that have a high public
  1571. like data sets that have a high public interest value but the cost is is really
  1572. interest value but the cost is is really
  1573. interest value but the cost is is really sometimes an issue and then you know
  1574. sometimes an issue and then you know
  1575. sometimes an issue and then you know even though we are hackers and coders
  1576. even though we are hackers and coders
  1577. even though we are hackers and coders you know we're not ai specialists right
  1578. you know we're not ai specialists right
  1579. you know we're not ai specialists right you know and so
  1580. you know and so
  1581. you know and so knowing the right thing to do is often
  1582. knowing the right thing to do is often
  1583. knowing the right thing to do is often kind of a trick absolutely so if if
  1584. kind of a trick absolutely so if if
  1585. kind of a trick absolutely so if if people want to know how to support you
  1586. people want to know how to support you
  1587. people want to know how to support you or big local news how can they do so
  1588. or big local news how can they do so
  1589. or big local news how can they do so well i mean we got all this code that's
  1590. well i mean we got all this code that's
  1591. well i mean we got all this code that's open source you know the warren project
  1592. open source you know the warren project
  1593. open source you know the warren project i just talked about briefly has more
  1594. i just talked about briefly has more
  1595. i just talked about briefly has more than a dozen outside contributors and so
  1596. than a dozen outside contributors and so
  1597. than a dozen outside contributors and so if you're interested in writing a web
  1598. if you're interested in writing a web
  1599. if you're interested in writing a web scraper that's part of one of our
  1600. scraper that's part of one of our
  1601. scraper that's part of one of our projects or just getting involved and
  1602. projects or just getting involved and
  1603. projects or just getting involved and improving some of our open source code
  1604. improving some of our open source code
  1605. improving some of our open source code there's a lot of opportunities on the
  1606. there's a lot of opportunities on the
  1607. there's a lot of opportunities on the github page just to jump in and grab a
  1608. github page just to jump in and grab a
  1609. github page just to jump in and grab a ticket and do it um also like you can
  1610. ticket and do it um also like you can
  1611. ticket and do it um also like you can always just reach out to me and i'd be
  1612. always just reach out to me and i'd be
  1613. always just reach out to me and i'd be happy to plug in this way or that way um
  1614. happy to plug in this way or that way um
  1615. happy to plug in this way or that way um in terms of some of our more the bigger
  1616. in terms of some of our more the bigger
  1617. in terms of some of our more the bigger projects that i was talking about if
  1618. projects that i was talking about if
  1619. projects that i was talking about if people want to are interested more in
  1620. people want to are interested more in
  1621. people want to are interested more in the big think of it or how we go about
  1622. the big think of it or how we go about
  1623. the big think of it or how we go about the bigger issues just go ahead and
  1624. the bigger issues just go ahead and
  1625. the bigger issues just go ahead and reach out you know i know that there's a
  1626. reach out you know i know that there's a
  1627. reach out you know i know that there's a lot of interest in the tech world and
  1628. lot of interest in the tech world and
  1629. lot of interest in the tech world and police accountability and some of these
  1630. police accountability and some of these
  1631. police accountability and some of these other stories and you know your talent
  1632. other stories and you know your talent
  1633. other stories and you know your talent and your knowledge is something that we
  1634. and your knowledge is something that we
  1635. and your knowledge is something that we could probably learn from so would be
  1636. could probably learn from so would be
  1637. could probably learn from so would be happy to really talk with anybody
  1638. happy to really talk with anybody
  1639. happy to really talk with anybody awesome
  1640. awesome
  1641. awesome then unless you have anything else you
  1642. then unless you have anything else you
  1643. then unless you have anything else you want to share i really really appreciate
  1644. want to share i really really appreciate
  1645. want to share i really really appreciate you coming on uh prefix live and
  1646. you coming on uh prefix live and
  1647. you coming on uh prefix live and anything else for for the crew out there
  1648. anything else for for the crew out there
  1649. anything else for for the crew out there no i mean thanks for having me um thanks
  1650. no i mean thanks for having me um thanks
  1651. no i mean thanks for having me um thanks for creating kind of a positive
  1652. for creating kind of a positive
  1653. for creating kind of a positive community thanks for putting up with my
  1654. community thanks for putting up with my
  1655. community thanks for putting up with my coughing i had i had a non-covered flu
  1656. coughing i had i had a non-covered flu
  1657. coughing i had i had a non-covered flu last week which i think i'm mostly over
  1658. last week which i think i'm mostly over
  1659. last week which i think i'm mostly over but i'm not quite there yet absolutely
  1660. but i'm not quite there yet absolutely
  1661. but i'm not quite there yet absolutely well thanks for being a part of the
  1662. well thanks for being a part of the
  1663. well thanks for being a part of the community and i think you
  1664. community and i think you
  1665. community and i think you along with everyone else but you have
  1666. along with everyone else but you have
  1667. along with everyone else but you have helped to contribute to that positive
  1668. helped to contribute to that positive
  1669. helped to contribute to that positive community so that's that's you for sure
  1670. community so that's that's you for sure
  1671. community so that's that's you for sure too so thanks everyone for watching um
  1672. too so thanks everyone for watching um
  1673. too so thanks everyone for watching um and we'll see you soon see you ben
  1674. and we'll see you soon see you ben
  1675. and we'll see you soon see you ben thank you

Downloads

Recording video · Timestamped transcript