- O-N-A-L-A, LA Times Meetup, if you're tweeting this,
- O-N-A-L-A-T hashtag.
- Welcome, we're, you know, this is kind of new ground for us.
- We're, the LA Times is happy to host this event,
- to get together more digital journalists in a room
- than in this town, than we've seen in a while.
- And I just welcome you and haven't prepared any remarks,
- so I'm gonna turn it over to Julie,
- who perhaps has prepared.
- Yeah, there's this idea it's gonna get better from here,
- but you might be disappointed.
- Thank you guys so much for coming.
- I really, I have no idea who you all are,
- and that's so exciting.
- I just, we wanted to start doing more of these
- types of things to get people together,
- with wine and snacks, and some information.
- So we hope to keep doing these things,
- and actually Robert is gonna talk about that.
- Hello, I'm Robert Hernandez.
- My Twitter handle is webjournalist,
- some people know me from that.
- Stand over here by the screen.
- We, Kim and I started O-N-A-L-A about two years ago or so.
- I'm a professor at USC, I moved back to LA
- about three years ago, and I wanted to meet other web nerds,
- and my virtual office wife and I met.
- And since then, we've been trying to get people together
- to kind of connect and nerd out and support each other.
- And normally it tends to be with like 48 hours notice
- at the Redwood, we haven't seen each other in a while,
- so we meet and drink.
- And thanks to Julie, we have this formal presentation
- or programming that we hope to do every couple of months.
- We wanna do it more, you know,
- the formal sit down learning,
- mind exchange every couple of months,
- and then some social meetup every other month
- or something like that.
- It's great to see so many folks,
- I know a chunk of you are from LA Times,
- and some new folks I don't know as well.
- Hope that this becomes a more routine thing,
- because it's better, journalism is better,
- the community is better served
- when we kind of share our knowledge,
- and experience, and be nerds together.
- Thank you for coming.
- Hi, I'm Kim, I am Robert's virtual office spouse.
- And my only job is to tell you that if your newsroom,
- or if you have ideas on what we should do for this programming,
- we are willingly accepting them.
- Just find one of us before, after the programming,
- pitch your idea, or if you have a cool bar we should go to.
- And also become a member of ONA,
- because it's kind of awesome.
- Who is a member of ONA?
- I'm on the board for the national.
- So, ONA, if you don't know, tends to be DC focused,
- or historically it's been East Coast focused.
- I got elected onto the board almost two years ago,
- I'm up for reelection if I decide to run again this year.
- But one of the things that I wanted to do
- when I was on the board was to make sure
- West Coast was represented in some form.
- But we need members to participate.
- Outside of a great conference,
- there's other things like this
- to kind of justify your membership,
- but at the very least, as an ONA person,
- it's this group that is growing.
- I think it's the largest international web journalism,
- online journalism organization.
- And it's just growing and getting better,
- but only made better through its members.
- So please consider joining.
- I think it's like 75 bucks or something, go on.
- Please.
- Martin's, sorry guys.
- Martin's gonna introduce the wonderful data desk people,
- but if everyone could give Martin a hand
- because he did a great job getting all of this together.
- Thank you.
- So, without any further ado whatsoever,
- Megan Garvey is our,
- she's somehow attached to the data desk.
- No one's really sure, but she'll tell you more about it.
- She provides the alcohol.
- I do do that, which is probably
- why anyone listens to me at all.
- They pointed that out to me.
- So I'm Megan Garvey, and I've been at the LA Times
- since 1998.
- I started out as a very traditional print journalist,
- and I can't even remember the first time I saw our website.
- And then a few years ago,
- I proposed an idea to do a database of California's war dead.
- So the men and women who'd been killed
- in the wars in California.
- And at the time I'd worked with Doug Smith,
- who's sitting, Doug, raise your hand.
- There he is.
- I had worked with Doug Smith on some data oriented projects
- in the past, but at the time there was really no way
- to publish most of what we did online.
- So a lot of work went into stuff that ended up
- as print graphics or informed stories,
- but was never really made accessible to the public.
- So I went to Doug and I said, I have this idea.
- And he said, two months ago, I would have said,
- great idea, no way, we can't do it.
- But we just hired someone named Ben Welsh,
- and I think you should meet him.
- So that was the beginning of, I don't know what it is, but.
- A beautiful friendship.
- Yes, a beautiful friendship.
- So the data desk here is really not,
- it's funny because a lot of people think of it
- as an official thing, but really what it is
- is a group of people from different sections of the paper
- who've kind of banded together.
- I don't know if it's on a life boat or a ship or raft
- or I don't know, a parachute.
- I'm not sure exactly what it is, but.
- Yes, only Doug, some kind of motorcycle.
- But the nice thing about that is that we work together
- by choice, and so it's been a lot of fun.
- And we have a weekly meeting we call show and tell
- where we often argue and fight with each other,
- but at the end of the day, come up with good ideas.
- And Ben is gonna come up and talk more
- about what it is that we actually do
- and what the data desk is.
- And then we're gonna take a look at a project
- that actually was born out of a lot of fighting
- and arguing and discussing.
- So Ben, come on up.
- Hello, okay, so does this work when I stand this far away?
- Can everybody hear me?
- Okay, good.
- My name's Ben Welsh, I work at the LA Times,
- and I do this stuff like Megan said.
- And our little band of people has been together
- for a few years and we've put together some stuff,
- and I'm gonna run you through kind of what we do
- and our work, add some examples of that,
- and then hand it off to Anthony to get more specific, okay?
- So I'm gonna go through a lot of stuff really fast,
- and if there's anything you have questions about
- or you wanna know more about,
- just raise your hand or say, hey, Ben, or whatever,
- because this is informal enough that we can stop
- and have a conversation, at least I hope so, okay?
- So feel free when the time comes.
- So the name of my talk is What is a Data Desk?
- And I'm gonna walk you through kind of what we do.
- If you wanna check out the slides later,
- they're at this URL here at the bottom,
- which is lat.ms slash Who is Data Desk?
- Okay, so like Megan said,
- the Data Desk is an informal team of reporters
- and programmers here in downtown LA at the LA Times,
- emphasis on team.
- And what we do basically in like a really oversimplified way
- is we turn databases into news,
- though in some cases, news is into databases,
- but that's like a whole other thing.
- But usually we turn databases into news.
- So like an example would be the California War Dead database
- that Megan talked about where there's a record
- for every casualty and then all those go,
- every one of those casualties gets its own page
- on the internet and you sort of turn the database
- into something on a news website
- that is different from a story, right?
- Other examples we've done that are more notorious
- include the rating of how well every teacher at LAUSD
- improves the test scores of their students,
- every person who's on the Hollywood Walk of Fame
- would be another example too.
- This is where you build the database
- that is sort of news in its way, right?
- That means that we write code, right?
- Like real code, we don't just talk about it,
- we do it and we do it all day, right?
- And I like to write a lot of Django, that's my favorite,
- and we do a lot of our projects in Django,
- which is a Python framework
- that makes it easier to build websites.
- So it's the Python programming language,
- it's all these handy shortcuts
- that make doing stuff a lot easier, right?
- And this is an example of a stack
- that you would use to put out a Python website
- that actually a lot of our sites
- kind of work a little bit like this.
- But that's not all that we use, right?
- There's a lot of other technologies that we use,
- there's people different from me
- who have different opinions and use different software
- like Doug, who uses a lot of SAS,
- though I'm trying to win him over, gradually.
- And there's other projects that you work on
- where certain tools are better than other tools, right?
- And so these are just some examples
- of different technologies we use.
- I'm sort of, I'm showing my age
- with the DHTML reference, right?
- Is anyone here old enough to remember that, right?
- Yes, right, I like to remind the younger developers,
- the internet had buzzwords before you were born, right?
- And anyway, but we also still can write in English,
- we didn't lose that when we became computer programmers.
- I know some people think that or that like they're somehow,
- they're like two, the two things don't go together,
- but that's not true at all.
- We remain the people we were before,
- we learned to program, right?
- And these are some examples of some front page stories
- that have come out of our work, right?
- And so now here we go through it.
- One thing we do, we make a lot of maps, right?
- That's the globe lobby you've just seen.
- I call it Art Deco Data Vis, right?
- In a certain sense, but we have a site
- where we map out the most recent crimes
- in the city of Los Angeles by neighborhood.
- Ken, it built a beautiful site to show every homicide
- in LA County over the last what, six years?
- Something like that.
- Something like that and Malloy and Sarah Artilani
- do a ton of awesome work to keep that up to date.
- You know, there's, we do a lot of other data
- by neighborhood, this is the census,
- this is every marijuana dispensary in Los Angeles, right?
- And we're putting out a lot of maps like this.
- Yeah, I know, this is not minimal, right?
- This is whatever the opposite of minimal is.
- And this is a very recent map I'm geeked about
- because this is one that Ken did to coincide
- with the LA riots with a lot of research from Malloy
- that shows everyone who died during the LA riots.
- And one that's really interesting in itself,
- but one little geeky thing
- that maybe some users wouldn't notice
- is this is the first LA Times map
- that has custom map tiles.
- It doesn't rely on Google maps for its service.
- Ken has done a lot of awesome work
- using OpenStreetMap and Tile Mill
- to get us on the path to escape from Google maps, right?
- And so this was our Sputnik.
- So besides maps, we also do investigations, right?
- You know, a lot of data work that people think of
- as data journalism or as this brand new web thing
- that never existed before because whatever,
- because big data is a buzzword now, it isn't true.
- I mean, computer assisted reporting is a geeky term,
- but it's been around for 25 years
- and people like Doug and Sandy and Malloy
- who are in this room have used technical skills
- to do investigations for literally decades
- and we still do that, right?
- So a recent example is Malloy and Ken
- worked on a story about all those people
- who then died in the riots
- and the cases that are still open, right?
- That's a recent one.
- Last year we did a big investigation about autism
- in California and across the country
- that included a lot of data analysis
- done by Doug and Sandy Poindexter
- who I don't think is here.
- But that's really good stuff if you haven't seen it.
- That's at latimes.com slash autism.
- I'd recommend the whole package.
- And then, you know, when Occupy hit last year,
- I was able to use some data to do a quickie census
- of the people who were arrested, right?
- So we still do a lot of these classic things
- that have to do with data analysis and investigations.
- Right, we also train robot reporters.
- That's another thing that we do, right?
- So what does that mean, right?
- Well, so for instance, this is a blog post
- from the Homicide Report, right?
- Which is a site that Ken built
- and is staffed by some people here at the Times
- where we write a blog post for every person
- who is killed in Los Angeles County, right?
- We're the only ones who do this.
- Now, you know, that's a lot of work and we can't staff it.
- But one way to make it less work
- is to automate some of the easy stuff.
- So as soon as the spreadsheet comes in from the corner
- every Monday, every Monday, Malloy, right,
- feeds it into the system.
- And the system, just based on the basic data fields
- that are in the spreadsheet, the age of the person,
- their race, their name, the location of where they died.
- Is there anything else that's, how they died
- that like you ever stabbed or shot?
- Ken has written some code that will take those rows
- and fields from the spreadsheet
- and write the first paragraph to every story.
- So before any human has to do any work,
- well, barring Malloy, you know, right?
- Right, sorry, Malloy's going by pseudonym tonight,
- I'm sorry.
- That initial paragraph is automatically written
- with no human intervention.
- And that's not all we do, but it's the bare minimum, right?
- So we have something up as soon as possible.
- And then if or when we're able to assign more resources
- to cover homicide, that person can then write through that
- and add more material onto the post, right?
- Which in this case was Robert Lopez,
- who was in the room but had to leave, right?
- And we take that approach to a lot of different things.
- Here's a blog post that appears a couple times a week,
- every time new crime data is fed to us by the LAPD,
- where we have an algorithm that goes through it,
- identifies the neighborhoods that are having unusually
- high amount of crime, and writes this post
- and makes this map and does all this
- with no human work whatsoever, right?
- We have pages for all these neighborhoods about crime
- and all this text is automatically written by algorithm.
- And one, the example I think is the coolest
- is one that Ken did, where he has a computer application
- that sits on top of the USGS's database
- and notification system.
- And every time an earthquake notification goes out,
- it processes that, has certain filters it puts it through
- before it decides it's newsworthy, right?
- And that's his news judgment written into code, right?
- And then if it's a newsworthy earthquake
- according to the algorithm, this blog post,
- this map, all this automatically written, fed into the blog,
- email sent to the editor, get on it, right?
- I bet people in this room have covered earthquakes.
- Somebody has probably, who doesn't work here?
- And what, yeah, and what usually happens, right?
- What usually happens, either it hits the wire
- and you're like, oh shit, we missed it.
- Or somebody says, did you feel that?
- And then everybody is clicking around on the USGS website.
- Where is it, where's that link?
- They have 18 versions.
- Which one is this one, it fought, fought, fought, fought, fought.
- Right?
- And in this case, in our case, that's Ron Lynn, right?
- That's the, for the LA Times people.
- But we don't need to do that anymore.
- The computer can do it.
- And Ron Lynn could work on his investigation
- into corruption at the Coliseum Commission
- and freakin' make our city a better place to live.
- You know what I mean?
- You worry about this stupid earthquake, right?
- Unless an earthquake hits the city.
- Unless an earthquake hits the city.
- One that surpasses a certain barrier on your algorithm, right?
- So anyway, that's a cool example of that.
- We also build open source tools, right?
- We have a GitHub account.
- Who here is on GitHub?
- Right, so GitHub is like the social network
- for computer programmers.
- It's where you put your open source code up on there
- and your friends can follow it, right?
- And when you make an update, you patch some code,
- they get a little Facebook update, right?
- And you can, it's true.
- It sounds, and it's awesome.
- So I don't know what's so funny.
- But like if you wanna like learn to code
- or get into programming or have something better
- than a crappy PDF resume, I'd recommend
- that you put your code up on GitHub
- and share it with the world and you know,
- start participating in what is really a historic
- like thing in computer programming.
- And so we have a GitHub account and you know,
- we're not anything special, but we've got you know,
- 10 or 12 open source projects that are out there
- that we're you know, trying to build a community around.
- And this is just a couple examples.
- One is we have a framework for quickly making
- an interactive table.
- You know, you wanna make, you wanna search and sort
- and filter really simply a table on deadline.
- We've built a framework internally that can do that
- and open sourced it.
- It's called Table Stacker.
- It's being used by about a half dozen different newsrooms.
- This here on MinPost.
- Here we had, Ken made a really great library,
- a wrapper on top of the AP's election data service,
- that like FTP that they dump everything on,
- that makes it really, really easy for you
- to write Python code to put the data out
- and put it up on your website.
- And after he did that, probably what,
- six or seven different news organizations
- have just stopped writing that same redundant code
- themselves and instead are using ours.
- And not only are they getting something out of it,
- they're contributing patches, fixes and features
- to our code so we don't have to do any work
- and they're making our life easier, right?
- Okay, so that's the benefit.
- Right, another example is this is a fun one.
- It's like auto complete for address searches.
- It uses the Google Geocoder as you type in an address.
- That's the LA Times address.
- It starts to suggest different places, right,
- which is kind of fun.
- But there's a lot of other ones if you go to
- github.com slash data desk.
- Okay, and Anthony who's gonna follow me up here,
- this is him on Twitter, Anthony J. Pesci, so follow him.
- He does a lot of different work here,
- but one thing that he focuses on
- and has really had a lot of success with
- is taking some of our print or less interactive
- graphic stuff and making it more interactive
- or more up to speed with the latest stuff on the web.
- And he's gonna show you in depth one of those examples,
- but here's some other stuff that he's done.
- We have a template now to make a California counties map
- just using a spreadsheet.
- We just upload the spreadsheet,
- boom, we've got the map like in 10 or 15 minutes,
- the power of automation, right?
- These are maps we have to do a lot.
- Same thing for the US states, he's made that.
- This is a recent one that he did for a big Sunday package
- that's more custom, that's about the boulevards
- of Los Angeles, our architecture critic is walking people
- through the major changes on them.
- And this sits on top of this big story he's written,
- which you can find where?
- What's the URL?
- LAtimes.com slash boulevards.
- This just came out on Sunday.
- We have templates now for doing these big long graphics
- where you sort of have a little nav that changes
- as you scroll down and you can link to.
- And the template is done in a way so that the graphics staff
- who couldn't build a page like this without some training,
- all they have to do is upload the images at a certain size
- and boom, they've got a graphic like this.
- So since they've done this,
- since we introduced this template,
- they've probably done what a dozen of these?
- Right, and it takes graphics work that in many cases
- either would have been in print,
- appreciated by print readers and then hated on the web
- because it was in a crappy template
- or just totally lost on the web
- and puts it in a really just simple template
- that makes it like actually halfway enjoyable
- for the internet, more than halfway Anthony, like, you know,
- like, but yeah.
- And it's really been amazing to me to see the response
- on social media where it's just graphics
- that were in a sense routine
- for our really talented graphics artists
- that were just ignored online,
- suddenly can have a lot of success on social media
- just because there's something to link to
- that people don't hate, right?
- And so like this particular graphic,
- this is the, of all the things we've done
- since I've been here,
- one of the things I'm most proud of
- is that William Gibson tweeted this graphic.
- The man who like freaking brought drones
- and science fiction together
- and like was way ahead of all this stuff,
- he loved this graphic, right?
- Which was by Raul, I think down in the graphics department.
- Yeah, see, thank you, Will.
- Will works down there.
- And you know, would William Gibson have seen it
- and tweeted it to his tens of thousands
- of really influential like tech followers
- if it hadn't been in something, you know, reasonably good.
- Also, Anthony did a lot of great work
- in getting live data out of Ken's data feed
- and into some interactive maps
- so that we were able to have live county by county results
- for all the GOP primaries, right?
- Anyway, so you should follow us, right?
- And here's where you can find us on the web.
- Our website where we have our big projects
- and also a feed of our latest stuff
- is at datadesk.latimes.com.
- You can follow us on Twitter at latdatadesk,
- which will have all that latest stuff as well.
- And you should get on GitHub if you're not already
- and follow us at github.com slash datadesk.
- That's about it.
- We consistently do have internships here at the LA Times
- so if there's anyone who would be interested with that
- in the last three or four years
- we've had three data desk interns, is that right?
- The first was Ken who now works here.
- The second was Michelle Minkoff
- who now works for the AP in DC, right?
- And the third was Alan Vestal who is either gonna take a job
- or go to grad school, I haven't heard yet,
- but has been offered jobs.
- And so there's a lot of demand for this stuff.
- If you wanna pick up some skills
- and get thrown in the deep end, we're happy to do it.
- And I promise you there'll be work for you somewhere
- when you come out the other side.
- So if you wanna do that.
- Apply with a GitHub account.
- Apply with a GitHub account and talk to Dan Gaines
- who's my supervisor who I don't think is here
- but his email is very small at the bottom.
- It's daniel.gaines at latimes.com, right?
- Or onlinejobsatlatimes.com
- Or onlinejobsatlatimes.com says Tracy Boucher
- who you could probably also bother after this meeting
- if you wanted to.
- And that's my whole talk.
- Does anybody have any questions
- before I hand it over to Anthony?
- Robert.
- Do you take two seconds not to buy a Kickstarter project?
- Okay, so outside of my work in the last few weeks
- I launched a website which is called Past Pages.
- It is at pastpages.org.
- And it is essentially, let's see if we can fire it up here
- maybe, let's run with scissors as Sarah Cohen would call it.
- We're gonna use the live web.
- And essentially every hour it goes through
- several dozen internet news site homepages
- and takes snapshots of them
- where it will hopefully archive them forever.
- A lot of you are probably familiar
- with the Wayback Machine at archive.org
- which is a really awesome site.
- However, it doesn't update often enough
- or regular enough to really do thorough study
- of what media is up to.
- And so the goal of this site is to really regularly capture
- as many media homepages as possible
- so that academics like Robert can have a great resource
- to study what happens on these pages over time.
- Like what if you could do an academic study
- of every photo of Michelle Obama
- that was used by the media before this election, right?
- Or whatever, right?
- You can start thinking about different ways you can do it.
- Or when we have a financial collapse next year,
- how the Wall Street Journal decides to cover that, right?
- So it has an academic use.
- Second, hopefully I think it has a media criticism use.
- For instance, last week the Drudge Report
- posted a false story about how President Obama
- had fabricated things in his memoir.
- It was only up for several minutes and then it disappeared
- but the screenshots on this site captured it.
- So if someone wanted to write a story pointing out
- that Matt Drudge likes to hype things
- that usually aren't true,
- they could use that as evidence.
- Right, so, and also, you know,
- if you work at one of these news outlets
- and you just wanna keep track of what's going on
- with your homepage over time,
- I hope it can be useful to you.
- This is an indie endeavor that I have funded
- out of my own pocket and built on my own time.
- But thanks to everybody on Twitter and other people
- in the awesome community of online geeks,
- I've raised $5,000 via Kickstarter to try to keep it alive.
- And so if you have any thoughts about
- what it ought to do or ought to be
- or what I've screwed up with it,
- please feel free to tell me afterwards.
- It is pastpages.org.
- Any other questions?
- No, okay, well, with that, I'll hand it over to Anthony.
- This is one of the more recent projects that I've worked on
- and it is a sort of interactive web graphic
- for looking at how super PACs are spending their money
- in the primary race.
- Does everybody know what a super PAC is?
- So a political action committee is sort of
- supposedly independent organization
- that can spend money during a campaign, right?
- Super PACs, a result of a recent Supreme Court decision,
- can spend unlimited amounts of money
- called independent expenditures
- and can receive unlimited donations from interested parties.
- In many cases, super PACs are not supposed to be
- tied to a candidate or controlled by a candidate in any way.
- They're supposed to be independent,
- but many of these PACs are run by people
- who are friends and advisors and work with
- some of the candidates.
- So we've tried to describe some of the super PAC activity
- happening in the selection using this graphic.
- Before I get into it too much,
- I wanted to show ProPublica has done some excellent work
- looking at super PACs also, both spending and contributors
- and the New York Times has done some work
- on super PACs as well,
- looking at some of the top contributors among other things.
- We took a look at some of this work
- and decided that we wanted to contribute something
- to the story about what was happening with super PACs
- in this election season.
- And we sort of sat in a meeting on a Friday afternoon
- and argued for about an hour.
- And sort of decided that, after looking at some of the data
- and decided that what's probably going to be
- the most interesting story or at least a very interesting
- story is how are these candidates using affiliated
- but not affiliated PACs to spend money on other candidates?
- And we sort of discovered, tell you right here,
- 56% of all of these expenditures
- have been spent opposing candidates.
- And you can see from looking at this graphic
- that Mitt Romney spent an exorbitant amount of money,
- almost $21 million opposing Santorum
- and almost $19 million opposing Newt Gingrich
- and in large part knocked them out of the race.
- So you don't really get that looking here.
- You don't really get that looking here.
- And I think one of the key things we decided
- both specifically about this graphic and in general
- about the kinds of presentations we wanna do
- for a database is it needs to tell a story.
- So for us it was really important to have something
- that was quick and relatively digestible
- where you could come here and see what was happening
- in a way that you couldn't if you just received
- sort of a dump of a database onto a webpage.
- And you can dive in.
- We have a timeline where you can look at spending over time.
- So you can see Romney's really concerned about Newt Gingrich
- in January and February.
- Newt didn't do so well.
- He starts contributing against Santorum.
- Most of these large jumps you see by the way
- are right before key state primaries.
- And then, okay there goes Santorum
- and lately now he's spending money on himself.
- You can see very recently Obama has started spending
- some money against Romney.
- We still have people like Herman Cain and Huntsman
- and Rick Perry here.
- But you can dive in.
- So it tells a story initially when you come to the page
- to look at it and it allows you to sort of come in
- and explore and look at how the spending has evolved
- over the course of the primary season.
- Any questions so far?
- Okay.
- We also have a spreadsheet available of the biggest donors.
- Very, very wealthy and politically interested people
- who are contributing to one or more super PACs
- and in incredibly large amounts of money.
- This would not have been really possible before
- or at least not like this.
- So I'm gonna tell you a little bit about how we get
- and clean and process the data and store it in a database
- and then I'm gonna talk very briefly about some of the code
- that went into it.
- This is, we get all of our data from the FEC,
- the Federal Election Commission.
- If you come to this page, FEC.gov slash data slash
- independent expenditure dot do,
- you can look at the latest filings
- and this is updated what, every day or two more?
- It depends on the committee but it's on a 24, 48 hour cycle.
- They update each file, each committee file.
- So the FEC has kindly provided this interface
- where you can come and you can start to explore the data
- and you can download it here with a CSV as a CSV
- which is what we do which gets you this.
- Not a particularly handy format
- although you can sort of see what they have, right?
- So there's a candidate name and some sort of ID
- and another ID, a state maybe, which office it is,
- how much money, the date.
- There's a lot of data here.
- It's somewhat indecipherable what's going on.
- So we wrote a program to come to this page,
- download the CSV, store it into a database
- and from there, there's a little bit of a manual process
- for cleaning up the data that Malloy can talk more about.
- But essentially the problem is when you come to this page
- and you download the spreadsheet,
- if you were to do a sum on this expended amount
- on the spreadsheet, it wouldn't actually be accurate
- because you can amend an FEC filing.
- So if you file one report that says you spent 20 bucks
- on advertising, you can come back and you can say,
- oops, I forgot a line item.
- I actually spent 20 bucks here and five bucks there
- and that just gets inserted into the top of the spreadsheet
- with all of the new expenses and all of the old expenses
- and you kind of have to manually go through
- and figure out what's what and what amended what
- and that's this column, AMN underscore IND, right?
- And if it's an N, it means it has not been amended
- and if it's an A1 or an A2, it means it has been.
- So you kind of have to figure out
- where these expenses line up.
- And Molloy, do you wanna come up for a sec?
- So Molloy is our fabulous researcher
- who spends her days toiling away doing this.
- So go to the FEC.
- So the one thing you'll notice on this site,
- this is the FEC site where you can look at,
- this is an example of associated buildings
- and it's every day they file a report
- and then they file quarterly or monthly
- depending on what cycle they're on
- but you can see here that that filing amended
- the one that filed it.
- So back in Anthony's database,
- what I have to do is I have to say,
- I wanna keep the 77820 and I wanna get rid
- of everything else because if I don't,
- then I'm gonna do this double count
- and there's not a nice way to do it
- except manually look at the file, the spreadsheet
- and look at those 5D numbers and then come back in here
- and get rid of the ones I don't want
- and keep the ones I do want.
- And it doesn't take a lot of time
- but I have noticed before that I've not been careful.
- I thought, oh, I've gotten rid of everything
- and then I'm like looking at things
- and I've forgotten to eliminate stuff.
- It goes by pretty quickly.
- You can do it in a pretty quick span of time
- and get rid of the things you don't need to have.
- So this is sort of a peek at our admin for this site.
- It's powered by a database of all of these expenditures.
- The one thing back on this file that we're starting with,
- this is Super PACs working to,
- for the presidential campaign and congressional campaign too.
- So that's our next thing we're gonna work on.
- We're gonna look at how Super PACs
- are supporting congressional races in California.
- But basically the workflow on something like this
- we've set up is I have this script
- that can go and fetch the spreadsheet from the FEC,
- look at everything, store it in our database,
- but we can't publish that live right away.
- You need to actually take a minute, look at the data,
- fix any of the amended filings,
- make sure it passes the smell test
- and then hit a publish button.
- So we have an interface where in this admin,
- you click a button and it pulls the data from the FEC.
- When the robot has finished loading it into the database,
- it emails me and Malloy.
- We see if there are any amended filings,
- if there are any new PACs, if anything's really changed,
- it'll summarize this in the email.
- Malloy can come in, see all of the new amended filings,
- check them manually against this horrible thing
- on the FEC's website by searching for the,
- the identifier, check a few boxes,
- make sure everything's kosher, hit publish,
- and then the graphic updates.
- So it's really important on something like this
- where you're relying on an external data source
- that you have a robot going to download and process
- that you actually do have a human who sort of sits there
- and looks at it and says, okay, everything looks good,
- everything's lining up, we're gonna hit publish.
- I would strongly recommend that.
- Other people have made a mistake.
- It's just a reminder that for all the stuff
- you do programmatically we still need common sense
- and knowledge and thought and so that,
- we kind of joke around about how they just push a button
- but the truth is there's almost none of that.
- Almost nothing that we do is that simple.
- It either required work on the front end
- like Ken's earthquake program to decide
- what was newsworthy and what should we exclude
- or on this stuff which is very complicated
- and we've spent a long time really learning
- the political campaign process
- and brings a lot of knowledge to it.
- So I mean Ben kind of jokes about like a robot reporter.
- The point is to take care of the busy work
- and reserve sort of the brain power and stuff
- that really matters to produce the quality content
- out the other end.
- Exactly and so this is a sort of database view
- of that CSV I was showing you earlier.
- This is all of the information we get from the FEC.
- We have a separate admin where we can for instance
- tie a pack to a specific candidate.
- So we do all of that in the back and then we hit go
- and then it spits out this nice page
- that is actually powered by an extremely robust database
- that we've spent a great deal of time developing
- and cleaning and checking.
- Malloy and I have double and triple and quadruple
- checked the accuracy of all of this
- on at least a number of occasions
- and we keep like going back and doing it again.
- So we know it's good.
- And this is sort of how you would arrive at that.
- It's a Django database.
- This is a very simple pack model
- where it's the Sierra Club.
- They have their FEC unique identifier
- and if we wanted to tie the Sierra Club
- to any one of these candidates we could.
- For those of you who don't have the programming
- and research resources to build your own database,
- write scripts to automate pulling in FEC data
- and have a researcher who has years and years of experience
- dealing with the FEC to go in and clean it up.
- The New York Times handily has a campaign finance API
- that actually ProPublica uses for this presentation
- that if you want to start playing around
- with some of this campaign finance data
- I would highly recommend it.
- I haven't spent a great deal of time in here myself
- although I've talked to a couple of the developers
- who made it and they are very smart people.
- And you can for instance come in and get a JSON list
- of the most recent expenditures that looks like this
- through the New York Times campaign finance API.
- You need an API key.
- I'm pretty sure anyone in this room could get one.
- And it's a very nice representation of the FEC data.
- I'm not positive the New York Times cleans it up
- in the same way that we do although I would imagine
- that they do.
- Probably something to look into if you're interested
- in using it.
- On the front end, so I talked about the database
- and stuff on the front end, this is all just JavaScript.
- So for those of you who know a little bit of HTML
- there's actually not a lot of fancy stuff going on
- like this is just a table.
- Like it's an HTML table which is handy
- because you're dealing with a graphic
- that it has huge sums of money that are changing
- and growing every single day.
- And you have this opposed column on one side
- and a support column on the other side
- and you need to make sure that everything is fitting
- into 980 pixels and that the bars are all sized appropriately
- for the space you have and the amount of money
- they're supposed to represent.
- Django, which is the software we use sort of on the backend
- to develop this, has some very handy features
- that you can apply on the template
- that will automatically resize your divs
- based on a value and a maximum width, right?
- So in this case the maximum width is 980 pixels
- and your value is 20 million dollars
- and then your maximum value is 33 million dollars,
- 34 million dollars.
- So like the pixels on the div are actually
- completely representative of the amount of money
- that they're supposed to have
- and everything sizes automatically.
- And all of that is sort of handled for me.
- The table adjusts automatically if more
- falls under the oppose or the support
- because an HTML table is just built that way.
- The divs all size appropriately
- because we have some code on the backend
- and a full featured piece of software
- to sort of handle that for us.
- And then, how many of you have heard of jQuery?
- Right, so if you don't know JavaScript
- or don't have a lot of front end development experience,
- you can actually sort of get a long way using jQuery.
- Like this, this was a feature we added later
- but we have the date picker and slider.
- Somebody had written a jQuery plugin
- for a slider by date range, or not by date range,
- just for a slider, right?
- And I went in and I lightly modified it
- to accommodate what I needed
- and in less than a day's work,
- it was working for my specific application, right?
- So chances are if you have a feature in mind
- or you want to sort of accomplish something technically,
- someone has already kind of done a lot of the work
- for you and open sourced it.
- And that was true here.
- So my focus here was writing some JavaScript
- to handle the math to sort of adjust all of these numbers
- and adjust the different sizes of the bars and the divs
- to accommodate the slide,
- but I didn't actually have to code the slider from scratch.
- So I got to focus on the actual problem,
- which was the math and making sure everything was adjusted
- and like the numbers and these tool tips update
- as you drag it, like, you know,
- making sure all of that worked as I wanted it to
- and tested that and not so much worry about the rest.
- We have some code on the backend also that takes, right,
- all of these different FEC expenditures and transactions
- we have stored in the database.
- It sort of looks at all of them in aggregate,
- does some math and spits out a JSON feed onto this page
- that we then use to render the graphic.
- Does anybody know what JSON is?
- Okay, it's a way of sort of serializing your data
- so that it comes out in a structured,
- easy to use way on the front end.
- Much like this, right?
- So instead of some crazy, right, like FEC spreadsheet,
- you actually sort of see the breakdown of the data
- and it's easy for you to come in
- and programmatically access and process.
- So that probably just about does it
- in terms of this presentation.
- I'm happy to answer questions about it
- or to show off some other stuff
- we've been doing with elections lately.
- So, yeah.
- Just the qualifying, if an ad is
- definitively opposing one candidate
- or supporting another candidate,
- because some apps do die and kind of split,
- even though the 30 seconds kind of split the time,
- is that an editorial decision that you see the FEC make?
- Luckily, that is not a call we make, right?
- So the FEC in their filing requirements
- makes them pick, right?
- So they have this neat column
- in their spreadsheet support repose.
- And that comes over to us automatically.
- Otherwise, it would be a nightmare.
- Like there would be literally no way.
- Yeah, that's an interesting question.
- I mean, maybe you could game the system
- by faking some of your FEC filings, although I doubt it.
- So the colors are key to the candidate.
- So this is Romney.
- This is Santorum.
- This is Mr. Gingrich.
- I mean, a bunch of us laugh because, I mean,
- colors are a horrifying job to get to online.
- You've got Anthony, God bless him, is color blind,
- so he's our special needs face
- on the data team.
- That's one of the reasons we kept him.
- He actually started as an intern on the onset before it,
- which he was not necessarily.
- And it taught himself, you know,
- most of that, all of his color stuff.
- So early on, when we had, you know,
- a large number of Republican candidates who were getting votes,
- we had to choose colors that would read easily online
- and not be shaped to their minds.
- Sorry, I was saying that early on
- when we had a lot of Republican candidates still in the race,
- we were faced with finding enough colors
- that were distinct and yet not political,
- you know, in order to display them.
- So for instance, like, you know, and we went,
- believe me, this was like through the ringer on this.
- This could be a whole ONA presentation on its own.
- But, you know, for instance, like,
- you can't give Romney red.
- I mean, that indicates that you think
- he's gonna win eventually, even if you did think that.
- You know, it's like, so we went through
- a lot of different incarnations.
- So all this is, is this was trying for consistency
- throughout our presentation.
- So as we were presenting the live election nights,
- live election results on primary nights,
- we wanted to make sure that throughout
- our whole presentation of the primaries
- and then into the general
- that we were consistent with colors.
- Of course, as we switch into the general,
- if, you know, Romney does in fact
- become the nominee at the convention
- and Obama is the nominee,
- then we go into much more traditional red and blue colors.
- But, you know, we have a whole,
- I mean, is this like the prettiest map we could have made?
- No.
- But can you understand who got which votes?
- And there's all sorts of things to like,
- you know, like there's this whole debate.
- Can we give Bachman pink?
- We had to switch that.
- You know, you're gonna make her pink, you know,
- and then, you know, you went right.
- So it was a big debate, but that's what it is.
- It's basically trying to be consistent from here
- throughout the rest of the products
- that we created for the campaign.
- I can tell there's probably other people in the room
- who have faced this problem.
- It's very, very fun.
- Any other questions?
- I wonder if how many LA Times journalists source this data?
- Also, do you see other journalists
- that are doing this data?
- All right, so the question is how many LA Times
- journalists are sourcing this data
- and what other organizations are doing the same thing?
- Okay, how many of us work on it,
- on elections or super PACs?
- Yeah, we actually field requests from reporters
- and our DC Bureau on a fairly regular basis,
- particularly with campaign finance.
- Malloy does a lot of that.
- Doug Smith and Sandra Poindexter
- also do a lot of that.
- I sort of have the keys to the database
- on the super PAC expenditures.
- So I help out from time to time
- when there's a super PAC specific question.
- So the number of people who are looking at
- and cleaning up and analyzing this kind of data,
- five or six maybe, including Malloy, Doug Smith,
- Sandy Poindexter, and then Ken and me
- and then other organizations.
- New York Times is doing it.
- Open Secrets does it.
- Anyone else?
- I'm sure there's other people.
- I think every lot of organizations are playing too
- because we all know that the kind of money
- that's going to be spent during this campaign cycle,
- why people have it, is kind of hard to wrap it all around.
- They're already very close to 100 million dollars.
- And that's not even talking about
- what the management committee themselves is going to do.
- It's going to be a lot of money spent.
- So our goal here was to try and get people an idea
- of how much money is being spent
- and how it is being spent.
- And you can see in this example here
- that we're on the test with the past affiliated with law
- and we are in support of the expense of a lot of money
- and we have to get rid of the
- I mean a lot of money.
- It was fascinating to watch with the timeline.
- You could see a moment in time when all of a sudden
- one of me turned his attention to the timeline.
- He no longer cared about the image.
- We could watch it and we got our interest in Europe.
- We were like, you just had a white coat.
- Now, you could see this moment in time.
- It was very clear that there had been a shift
- and that English was no longer a threat.
- For that, this has been really useful.
- I think it's really a good tool to try and explain
- how this campaign is going to play out with super PAC money.
- That was kind of our motivation in using it.
- I think a lot of other organizations
- are trying to use super PACs to tell the story
- and do it in different ways.
- Do you talk about, if you remember,
- what can you first, if you will,
- in terms of the branding, design,
- when you knew that the story was about spending a post-up,
- or did you go through data first and then say,
- this is the story we are going to tell,
- then you go to find it.
- Does that make sense?
- It does and it was a lot of both.
- We had looked at what some of our competitors were doing.
- We looked at ProPublica, we looked at MIT, OpenSecrets,
- a couple other people,
- and saw what they had done.
- It's, in some cases, a little bit difficult to come here
- and quickly process what's happening.
- We knew we wanted to tell a story.
- We didn't want to have a data dump
- that a reader or a viewer would have to come in
- and sift through in order to figure out
- what's going on in the world.
- So, we had this philosophy like,
- we don't have as many resources
- as some of the other news organizations out there.
- We're a small team, three developers,
- three people who analyze and play with data.
- We can't do everything and be everything all the time.
- So, a lot of the times, we want to try and focus
- on one really great thing,
- one really great part of the data that we can show off,
- one really great interactive or feature
- that we can do and do really well
- rather than trying to,
- rather than doing the whole thing.
- Like, you know, in this case, or in this case,
- like ProPublica has this really great tree map
- of the biggest donors that we haven't,
- we would very much like to do
- but have not gotten around to yet.
- This is another example of something like that
- where we just launched this recently.
- You know, the idea here was we wanted to give people
- information about what states were considered
- at play in the election.
- So, we have our wonderful DC Bureau
- wrote a very informative blurb about each state.
- And we've made editorial selections
- about which states are considered battleground states
- that the reader can come in and turn either Obama or Romney
- to see what the results could be.
- You know, and the Huffington Post and the New York Times
- and other news organizations have done a lot more
- with this kind of data.
- We wanted to make something that was both informative
- and fun, something that focused you on
- what was really the story, what was really going on,
- rather than giving you a ton of different options
- and sort of overwhelming the user.
- But, you know, so our philosophy on Super PACs was,
- let's tell a story, let's come in and find a unique way
- of presenting the data and telling the best story
- about what was interesting.
- So, we looked at what our competitors had done.
- We looked at the raw data.
- We sat around the table for a long time,
- sort of batting back and forth different ways
- of presenting it and different things to break it down by.
- And a lot of it was dictated by what the FEC gave us, right?
- So, they gave us support and oppose.
- We knew pretty well which PACs were affiliated
- with each candidate.
- And we decided we wanted to look at how
- those PACs were spending on other candidates.
- And we, you know, you kind of had an idea at the outset
- that it would probably be a lot of negative.
- And it was.
- And, you know, you come to this page
- and you immediately see that Rami has just spent
- exorbitant amounts of money opposing Santorum and Gingrich.
- And this was sort of,
- this kind of storytelling is what we were after.
- So, it was, you know, a combination of
- what's the best way of presenting the data?
- What do we have already that's available?
- And what looking at it do we think is interesting
- or is going to be interesting?
- Both, if the raw data already and just sort of knowing,
- having that knowledge of what we thought
- was probably gonna happen.
- I was just wondering if there was an exception
- in the equation, how many people
- were there to be required for the project?
- That's a good question.
- So, the question is from, you know,
- the first idea to launch, basically,
- how much time and resources and people
- were required for the project.
- I did pretty much all of the coding front and back.
- Malloy did a tremendous amount of work,
- initially cleaning up all of the FEC data
- and continually, every single day,
- every time we update it, it's coming in
- and doing work to clean up the filings.
- A few weeks.
- So, it was a pretty quick turnaround, considering.
- You talked about the secular sub-product,
- whether some of these people are using, you know,
- some of these people.
- Sure.
- Yeah, I decided to learn to code because I wanted a job.
- I, you know, I came here as an intern
- and I was a web intern and I got lucky enough
- to be placed with Megan on the data team, writing.
- Is it on camera?
- This is on camera.
- I was lucky enough to be placed with Megan Garvey
- as an intern.
- And sort of looked at Ben and looked at Ken
- when he came in as an intern also
- and sort of figured like, you know,
- the company's bankrupt.
- They're not gonna hire me unless I have
- seriously marketable skills, right?
- So, I worked on the skills.
- I started off, I sort of already knew a little bit of HTML,
- no JavaScript, so I polished my HTML skills,
- picked up jQuery.
- From there, I learned more JavaScript.
- Through books.
- Books and websites like W3schools is really good
- for HTML in particular.
- And then from there, I learned Python and then Django.
- Two years?
- Three years?
- Something in there.
- Oh, nine.
- Yeah.
- Since 2009.
- Takes a while.
- Yes.
- Absolutely.
- I think it's pretty rare that we launch
- a fully baked product right off the bat.
- And we're trying to sort of get away
- from that attitude as well.
- But no, this, it launched sort of as a more
- static version of this.
- The date range picker and slider was not here.
- But, like, this stuff was.
- Oh, right, this, right.
- The sort of rollover where you could see how a candidate
- get highlighted on it was added later.
- The date range slider was added later.
- But it launched looking a lot like this.
- And we'd like to add, like, it's nice to see the events
- that precipitated some of the spending.
- So like the South Carolina primary,
- or the Gingrich winning South Carolina,
- or St. John dropping out of the race.
- We've got some space issues.
- Anthony actually cleaned that up earlier today.
- So I think it's always a work in progress.
- I've never done that.
- And yeah, the other thing is we have this great database now
- of all of these expenditures and super packs.
- And we have one page, right, which is great.
- But it'd be nice if we had detail pages on each pack.
- It'd be nice if we could do a little bit more
- and tell a few more stories with the enormous amount
- of data we have and the great database we have from the FEC.
- So certainly there's a plan to do more.
- What it's gonna be yet is still a little bit up in the air.
- But this was an evolution.
- And then there will be more with the data later
- after the California primary, probably.
- Yes.
- Is there a lot of response to this?
- Or is there anything that you'd like to say?
- We've gotten some great response on this.
- Apparently not a lot of comments, but.
- But this map though, there are so many comments.
- It's really.
- But stuff like this, it does well.
- And I sit next to our homepage producers
- and can badger them to link to more interactive
- and database-driven work that we're doing.
- And we regularly are sending emails to all of our bloggers
- and reporters in DC to get stuff linked up
- that we do on their posts.
- So it's an ongoing effort to get even our stuff promoted
- across the website in the way that it should be.
- But when people do find it, it does really well.
- And people spend a lot of time on these pages.
- And they do well on their own, just in terms of page views.
- We have a much longer half-life story.
- The story's gonna get 99% views per two hours.
- Something like this can be linked to stories
- that come from the website.
- And then they actually have a lot more answers,
- but we're not always allowed to be linked to versions
- of the stuff like this under them.
- After all, it's gonna be linked to stories like this.
- Yeah.
- Sorry, so.
- The question is about how much of the work that we do
- makes it back into print.
- And Ben showed a couple examples early on of that
- where does this precise graphic make it into print?
- This has not yet.
- There might be other examples where it has.
- But I think that we've really worked to try to do more,
- to inform more of print by some of the web work
- that we're doing and not be so dictated
- by what we would do for print anyway.
- So that's definitely a struggle that has gone on.
- And the thing is, so like what Ben was saying,
- where there's a longer half-life, where this stuff lives on,
- where this stuff, you know what I mean,
- we're gonna eventually boil this down to Romney and Obama.
- And it'll change in that sense
- because the whole focus of what's going on
- in the news right now is about to change significantly
- as we look forward to November.
- But yeah, I mean that's definitely something
- that I think as an organization we're trying
- to think more creatively and think differently
- about how we produce the news
- so that the online stuff doesn't just stay online
- and the print stuff doesn't just stay in print,
- that there's much more communication back and forth.
- And I think Martin is ready to come up
- and bid you farewell.
- Thank you, Anthony.
- Very much.
- And we're gonna wrap up.
- We're gonna put the screen up.
- We're gonna open the bar again.
- And I just wanna thank you again for coming
- and join the ONA.
- I know I will, eventually, someday.
- Thank you, Julie, for the idea of this
- and for convincing us to do it.
- We had a good time and hope you all did too.
- And we'll be around, pinhole us, talk to us,
- follow us on Twitter, and so on.
- But thank you very much for coming.
- Thank you.
What is a Data Desk?
By Ben Welsh • • ONA LA in Los Angeles