Bare Facts First

How Reuters uses Datawrapper to create, automate and disseminate hundreds of charts each week

By • • Datawrapper in Streamyard

Slides

Show the extracted slide text

Slide 1

Bare Facts First
How Reuters uses Datawrapper to
create, automate and disseminate
hundreds of charts each week

Slide 2

Who we are
I’m Jon McClure.

I've been the EMEA graphics
editor at Reuters in London
since 2020. Before that, I led
“interactive news” teams at
Politico and The Dallas
Morning News.

Slide 3

Who we are
I’m Ben Welsh.

I joined Reuters last year after
15 years leading data projects
at the Los Angeles Times.

I’m helping our team create
more of what we call “news
applications.”

Slide 4

What is Reuters?
❖ Founded in 1851, Reuters is the world’s
  largest international multimedia news
  provider.

❖ Each day 2,500 journalists in nearly 200
  locations around the globe produce news in
  16 languages.

❖ Financial professionals receive exclusive
  news and data from Reuters via the LSEG
  terminal.

❖ More than 2,000 newsrooms republish our
  work via a wire service.

❖ Breaking news, analysis and special reports
  are available for free on reuters.com.

❖ A legally binding vow requires we uphold
  trust principles valuing independence,
  integrity and freedom from bias.

Slide 5

Our mission
❖ Breaking news has been at the core of our
  business from start.

❖ An 1883 memo spelled out the London
  office’s expectations for correspondents.

❖ At that time, stories were sent by telegram.

❖ The editors asked “that the bare fact be first
  telegraphed with the utmost promptitude.”

Slide 6

What you’re in for
❖ Jon on how the Datawrapper charts
  created by Reuters staff are distributed to
  the world

❖ Ben on how we created a software system
  that automates routine Datawrapper
  charts

❖ Your questions

Slide 7

Before Datawrapper
❖ High-caliber graphics team spread thin
  doing too much low-caliber work on
  request from other desks

❖ Too many good, simple charts not getting
  made because of the backlog

❖ Zero-sum between covering breaking
  news and doing significant enterprise

❖ Frustrated reporters resorted to a dark
  market of increasingly terrible and
  inscrutable charting tools

Slide 8

But before we could plug in Datawrapper, we had
to fit a pipeline that would serve our wire clients
as well as our website:

Slide 9

At Reuters, we publish each chart in many
different “editions” for other media clients to
pick from:

         Self-hosted and embeddable pages



         Static and editable images




         Editable source code




         Light & dark-mode charts

Slide 10

This is not the end. It is not even the
beginning of the end. But, it is perhaps the
end of the beginning, innit.

Slide 11

Using Datawrapper’s
          API                    publishing hooks and API with
                                 the cheap scalability of
                                 serverless computing, we can
                                 process, repack and publish
                                 those different charts editions.




Serverless lambdas process,
repack and publish our charts!

Slide 12

We use custom fields in the chart editor
      to include essential metadata we need
      to move charts through our publishing
      process.

API

Slide 13

Along the way, we flag new charts in
common Teams channels, which gives
our graphics team ample opportunities
to coach our newsroom.




     ✨Bonus: Need a Teams klaxon? Ours is free to use!
      reuters-graphics.github.io/teams-klaxon/

Slide 14

Within the last 18 months, Reuters
has onboarded more than 800
reporters, editors and producers
across our newsroom, who publish
dozens of charts each day.

That spread lets our graphics team
focus on higher spec graphics for
headline global news.

Reporters can publish more niche
charts on their local beat.

Producers have become frontline
chart makers for breaking news.

Slide 15

With our publishing pipeline in place,
graphics started developing automated
chart builders, linking external APIs with
pre-made chart templates at the speed of
breaking news.

“Tremolo” our USGS-powered earthquake
monitor published the first maps of the
deadly 2023 Morocco quakes within minutes
of USGS releasing the seismic data.

Slide 16

Publishing pipeline and a couple automated
chart builders underbelt, we were ready to
try scaling way up.

Slide 17

Text is automated
When Ben arrived at Reuters, he
learned it already has a text
automation system.

It’s called Lynx Insight. It fits
data to pre-defined templates
using carefully tailored logic.

This story is filed immediately
after new US employment
numbers are released each
month.

Slide 18

Why not graphics?
The graphics department head,
Matt Weber, long wanted to
automate graphics too.

As with Lynx Insight, it could
boost speed, increase volume
and allow highly-skilled talent
to focus on higher-skill work.

So Ben set out to try.

Slide 19

Feeding the funnel
Ben learned how Jon’s
distribution tools work.

His life flashed before his eyes.

Jon’s ingestion system for
Datawrapper was, in the
metaphorical parlance of Ben’s
tronc overlords, a funnel.

Ben just needed to figure out
how to feed it.

Slide 20

Enter the API
APIs allow programmers to
access online services with
code.

Datawrapper’s API allows users
to automatically generate
charts and maps.

All Ben had to do was send
instructions over the web with a
special access token.

Slide 21

Python is cool
Sergio Sánchez Zavala created
an open-source package that
lets you access the API using
the Python programming
language.

If you know Python, it’s easy to
use.

Ben knew Python.

Slide 22

Our advantage
Reuters is the exclusive news
provider for LSEG, a financial
service that competes with the
infamous Bloomberg Terminal.

That means we have access to
a high-quality, rapidly updating
feed of newsmaking numbers.

It’s the top source for Reuters
graphics, but reporters have to
claw out data files and paste
them into Datawrapper.

That’s slow.

Slide 23

Another API
Ben learned that the LSEG
system also offers an API.

He found its open-source
Python package.

He got excited.

Slide 24

The missing link
Ben realized that Python code
could download data from
LSEG’s API and then upload
charts to Datawrapper’s API.

All he needed was an “ETL”
pipeline that connected the two
systems.

So he wrote it.

Slide 25

The pipeline in theory

Slide 26

The pipeline in reality

Slide 27

For example
Here is Python code that —
using a set of custom ETL
helpers — creates a line chart of
the unemployment rate.

It’s just one small snippet within
a large system, but it shows the
basic bits needed for each chart
template.

Slide 28

Actions as platform
The system is run via Actions,
the powerful task runner
included with GitHub.

Yes, Actions is the production
environment.

Yes, it works.

Yes, it’s virtually free.

Yes, you should do this.

Slide 29

Down the funnel
After an Action makes a new
chart via the Datawrapper API,
Jon’s system picks it up and
does the rest.

The templated charts drop into
our graphics pool, just like the
custom charts made by our
staff.

Slide 30

Speed wins
The result is that when a
newsmaking number hits LSEG,
an automated chart can be
ready in minutes.

We’re seeing our charts land
even faster than our text
stories.

Slide 31

Scary scale
The templates can easily be
extended to sibling data sets.

This allows our system to scale
far beyond what any human
could do.

A powerful example is our bar
chart template for corporate
earnings reports.

It can instantly make a chart for
each of the thousands of stocks
tracked by LSEG.

Slide 32

Freedom rings
Ally Levine had to log on early
on Fridays to manually make
that unemployment rate chart.

Now the robot does the work.

Ally gets to sleep in and do
things like this instead.

Slide 33

Putting up numbers
We’ve been up and running for a
little more than six months.

Thus far, we have nearly 60
templates, including a few that
branch out beyond LSEG and
Datawrapper.

All told we’ve pushed through
more than 20,000 charts.

Slide 34

Merging back
Ben teamed up with Sergio to
expand and improve the
Datawrapper package for
Python.

It now covers 100% of API
features.

Slide 35

Python API code in reality

Slide 36

Python API code in theory

Slide 37

You must learn
Ben and Sergio created a
tutorial that shows newbies
how to use the Datawrapper API
with Python.

It was a standing-room only
event at the recent NICAR
conference in Baltimore.

It’s fully scripted and free to all
on Ben’s website.

Slide 38

Thank you!
Slides available at
bit.ly/bare-facts-first

Recording

Show the timestamped transcript
  1. our next two speakers. We are going to be hearing from John McClure and Ben Welsh, both
  2. from Reuters. John is the EMEA editor for graphics at Reuters in London, and he also
  3. works on their custom built pipelines that publish hundreds of data wrapper charts each
  4. week to a global readership. And Ben is the news applications editor at Reuters in New
  5. York City. He has led the creation of an entirely new system for automating charts with the
  6. data wrapper API. So they're here to speak on the automation framework that they use.
  7. It's the very first time that they're revealing this process. And in the interest of not delaying
  8. our peek behind the curtain, I will turn it over to John and Ben.
  9. Hey, thanks for having us. Thanks for coming, everybody. It's a real honor to speak at this
  10. excellent conference. I've been tuning in as much as I can over the last couple of days,
  11. and it's been super impressive and inspirational to hear. Should we get into it, John?
  12. Let's do it, Ben.
  13. All right. I think they already have the intros. They know who we are. We can buzz those. John,
  14. you want to tell people what Reuters is? Reuters is a newswire service founded in 1851. We're
  15. big place. So there's about 2,500 journalists in around 200 locations. What other factoids
  16. do you want me to pull out from all of that, Ben? What are you telling me?
  17. I mean, it's a huge place. I've only been here a year, and I've learned so much. And I think
  18. really maybe no one knows everything that happens at Reuters. But there's three kind of big pieces
  19. to the company that the outsider might not know. One is that service that John talked about
  20. that's so famous that sends photos and breaking news to more than 2,000 newsrooms around the world
  21. in all different languages. That's thing one. Thing two, which shouldn't be overlooked,
  22. is that Reuters is the exclusive news provider to one of the world's major financial terminal
  23. businesses, which has recently been rebranded as LSEG. It used to be known as Refinitiv.
  24. And that is, in many ways, kind of the Pepsi to the Coca-Cola of the Bloomberg terminal,
  25. if you're familiar with that. And then, of course, there's Reuters.com, which is where anyone can go
  26. in for free, read a selection of our breaking news, and our excellent investigative special reports.
  27. So those are kind of the three big pieces and arms of the company to keep in mind as we move through
  28. this. And at the core of all of it is really the lifeblood of the Reuters business, as it has been
  29. since the very beginning, which is breaking news, right? You know, literally, there are hundreds,
  30. if not thousands, of little itty-bitty stories that are flying out from our offices around the world
  31. every day. And it's been that way for more than a century. One of my favorite illustrations at this
  32. point is this very, very real 1883 memo sent out from the London office where John now works
  33. to the correspondents in the field, urging them to deliver more breaking news. I won't read the
  34. whole thing to you, though it is an entertaining read in general. One thing to keep in mind is back
  35. then the telegram was how breaking news was submitted. And the author of this memo is really pushing down
  36. on the troops to deliver more breaking news. Near the end, they actually give an enumerated list of
  37. the type of news that they would like to hear more of. My favorite little bit here, if you slow down
  38. and squint and read, is a call for all disturbances arising from strikes, duels between and suicides of
  39. persons of great note, social or political, and murders of a sensational or atrocious character,
  40. which maybe gives you a sense of how editors saw the news back then. It's still true to some degree
  41. today. But the key line in this memo that is quoted around the office quite frequently comes
  42. in that last paragraph where it is requested that the bare fact be first telegraphed with the utmost
  43. promptitude. And we say the line with a little bit of irony today, given the sort of elevated,
  44. you know, British diction, the king's English in there. But it really is still true and really the
  45. essence of our business. Being fast, being first, being simple and clear is really the primary
  46. imperative of the company. It's the company, we do so much more than that, but it really is at the
  47. heart and soul. And it's that kind of mission that we're going to talk about today and how Data
  48. Wrapper helps us serve it better. And we're going to do that one by John giving an overview of how,
  49. you know, he and his team built this, I think, truly wonderful system for distributing Data Wrapper
  50. charts across all those different arms we talked about. And then how I came in and tried to do a
  51. little, tried to do even more with it. So, John, you want to do part one? You want to take it, man?
  52. I'll take it. If you don't mind driving.
  53. So, like a lot of places, Data Wrapper filled a pretty predictable need for us in our newsroom.
  54. We are, I suppose, one of the larger graphics teams by comparison to folks who aren't working
  55. in some of the mainstay newsrooms, but still compared to the number of journalists that we
  56. work with, we're actually quite small. And so, for Data Wrapper, we had all the same problems,
  57. so I'm sure you've heard Echo Gravel's conference about, you know, being very thinly spread. It always
  58. seemed like we couldn't quite keep up with the reporting that the newsroom was doing and really
  59. sort of failing that first rung of our own company ethos to really be there for the big breaking news.
  60. And behind the scenes, the reporters and editors that we work with would often reach out if they
  61. couldn't get us to these really terrible kind of charting tools that would be sort of scattered
  62. throughout dark corners of Reuters infrastructure. I then found a very nice copy down there.
  63. I believe that is a real chart that's come out of one of the terrible systems. I think Icon is that one.
  64. And so, in a lot of ways, Data Wrapper was going to fill exactly the need that we all have in situations
  65. that we really need to scale out the number of people who can create charts in the newsroom for the next one.
  66. If you would, Ben, I think you're driving or am I, or can I drive, actually? Okay, cool.
  67. So, as Ben mentioned, though, we're a little bit different because of our wire service business.
  68. And one of the main tenets of that is that we have to service other media clients.
  69. And so, in addition to parking charts on a website, we actually also have to distribute them in a number
  70. of ways to other media who will then publish them on in their websites, papers, television segments,
  71. whatever, go ahead and pull, I guess I'll say. Or can I?
  72. I think you're going to have to drive, Ben. Sorry about that.
  73. So, at Reuters, we publish charts in a lot of different flavors for our media clients in particular.
  74. And so, we call these different additions, some of which you'll be very familiar with coming straight out of Data Wrapper.
  75. So, embeddable HTML pages. So, the chart that you can just embed with embed code.
  76. We also make those so that our media clients can host them on their own servers.
  77. But we also make static and editable images. So, PNGs, PDFs.
  78. We also make our source code for all of our graphics available to clients.
  79. We have clients who have developers in-house who like to customize things themselves.
  80. And then we do the usual kind of light, dark mode charts, little variations like that pull.
  81. But what that all means for us is that that little publish now button is not the end for us.
  82. Basically, that's just the start of a process by which we have to take our charts and make them ready for media clients.
  83. And to do that, we use Data Wrapper's publishing hooks in the API.
  84. Basically, what that does is it gives us the ability to, you know, suck out those charts and the various flavors and formats that we need.
  85. We can break open the HTML, CSS, and JS to do various things in them, repackage them the way that we need them to be packaged for our clients and ship them out.
  86. And that may sound like it's very specific to us, but I think it opens up a ton of options for anybody to do really, really cool things with Data Wrapper charts.
  87. We use AWS Lambdas so that this can really scale.
  88. We'll talk about how big it scales here in a second with Ben.
  89. But that post-processing step is really key to us being able to service our clients.
  90. You can go ahead and get your answer.
  91. We also use custom fields in the chart editor.
  92. It's a really easy way for us to attach some of the vital metadata.
  93. You'll see these roots and wild slugs.
  94. These are some of the weirdest nomenclature hats.
  95. I think these are basically little pieces of metadata that allow us to categorize and make it easier for the clients.
  96. And then once we've done all that, we've tagged our things, we've repackaged them, we've pushed them all up.
  97. We can ship them out to the world through a platform that we call Reuters Connect, which is where clients can come and purchase our charts in any one of the formats that I mentioned.
  98. As part of all that process, while we're doing all this repackaging, we take the moment to go ahead and flag charts that are coming, that our newsroom has made or that our team has made or that our bots have made.
  99. And especially in the context of our newsroom, it's actually a really great moment.
  100. And I think I've heard several people mention this and deal with Slack or whatever.
  101. It's a really great moment for us to coach our newsroom and really have sort of a feedback cycle about the charts as people are making.
  102. And this is one such example from some of the newsroom.
  103. And if you are, so at the end of the day, what that means is in the last 18 months, we've onboarded something north of 800 reporters, editors, and producers across our newsroom.
  104. They're publishing dozens of charts every day.
  105. And what that means is that the load is off of our, by comparison, small graphics, so we can focus on certain bigger things that we're known for, where we start to come back graphics.
  106. That also means that reporters can do more niche work in their individual beats.
  107. And our producers have actually become some of the most important chart makers we have.
  108. The first one is getting out maps, quick charts on the news as it's happening.
  109. So once we have that sort of pipeline in place, one of the things that we started to try to do was do some of the sort of maybe chart building that you've heard some of the others here talking about.
  110. This is our first example is done by Trebs Hartman in our graphics team.
  111. It's called Tremolo, which I think is a good name for it.
  112. It's a USGS powered earthquake monitor.
  113. So the, one of my favorite examples of this was last year, the Morocco quiz struck, we had the shape maps out within minutes, and we continued to pivot on those throughout the day, had more invitations to them, stuff like that.
  114. So we've now had this really great system by which we can, you know, either by hand from the newsroom or through an automated process, create a chart, and then publish it out into the world in all these various ways.
  115. And so we are ready to try scaling this thing way, way up.
  116. And that's where Ben comes in.
  117. Ben?
  118. Yeah, that's right.
  119. So, I mean, all this work was, you know, kind of ready and waiting for me when I started at Reuters, and I was trying to figure out what the heck am I going to do?
  120. Let's find something ambitious, a project to take on, something to chew on.
  121. And I found inspiration immediately from two places.
  122. One is a system that was invented here at the company a few years ago called Insight.
  123. And so this system is basically templated text stories.
  124. So when new data comes in on the feeds, they're sort of fit like a Madlib into a templated story and then sent directly out into the wire.
  125. And this is a type of system that exists not just at Reuters but at other news wires and has been very effective at increasing kind of the speed and scale of the text operation.
  126. The graphics editor, Matt Weber, you know, approached me and he said that he and others had had the longstanding idea of why can't we do the same thing for graphics, right?
  127. So if we can automate these Madlib text stories, why can't we automate the graphics that would go with the same thing as well?
  128. And I thought, man, he's really got a point.
  129. So let's look at it.
  130. Let's see if we can make it happen.
  131. And then at that point, I learned all about John's system, you know, which he just, I think, covered very well.
  132. And it kind of blew my mind.
  133. It's really just a powerful, you know, end-to-end kind of assembly line that's just waiting for widgets to kind of be pushed down it.
  134. Someone who can claim to have worked for the entire tenure of the Tronk company, it reminded me in some ways in my lighter moments of the funnel,
  135. which is one of the more hilarious, unintentionally hilarious metaphors in journalism history.
  136. If you haven't seen the video, the link's right there.
  137. Give it a look.
  138. And I just needed to figure out how I could get the data in there.
  139. So, you know, on the one end is the Data Wrapper API.
  140. If we could put charts into the Data Wrapper API, the top of the funnel, so to speak, we know they'll flow entirely down through John's system.
  141. And as has been covered by other speakers in this panel, that's totally doable, right?
  142. And then I'm a Python guy, not an R guy.
  143. I found, wow, there's this cool Python wrapper for the Data Wrapper API that makes it easy with a couple lines of Python code to pass in whatever chart you might want.
  144. Oh, okay, that's great.
  145. Started checking that out, started working with that.
  146. Sergio Sanchez-Zavala is the inventor of that, by the way.
  147. Cool guy.
  148. And then I saw on the other end, well, where are we going to get the data to put into the charts, right?
  149. Well, we've got this terminal, okay?
  150. So this LSEG terminal system really is this kind of incredible database that has limited access to a very small number of financial professionals who pay for it.
  151. But it updates literally within seconds of new market data being posted or macroeconomic indicators appearing on government websites.
  152. And learning about it has really been eye-opening and interesting to me as someone who's never worked at a terminal business before.
  153. The truth of it is, is behind a system like this is literally hundreds, if not thousands, of web scrapers that are crawling government data sites and putting in new figures instantly when they arrive.
  154. And then a whole quality control layer of people who make sure they flow through.
  155. So we had this great delivery system for graphics on the other.
  156. And we had this great data source, you know, on the other end.
  157. We just needed to connect them.
  158. Well, it turns out there's a Python API for the terminal as well, which would allow anyone with access to write code to download the data from the terminal.
  159. And all we needed to do was to kind of close the missing link between the data source, the terminal, and data wrapper, our delivery device.
  160. And that really was the work that needed to be done and was kind of the job for me if we were going to make it happen.
  161. And, you know, one way of thinking about a process like that, that kind of ferries data from one to the other, is what, you know, really gnarly, boring computer programmers call an ETL pipeline, right?
  162. I think it's good we're just calling them pipelines now.
  163. It's a little more human.
  164. But that's an acronym for extract, transform, and load, right?
  165. And what those typically do is the extractor, you know, downloads the source data from wherever it comes from, in this case, our terminal, right?
  166. There's some sort of computer programming layer that is going to transform, clean, summarize, reformat that data so that it's ready for delivery to kind of its target.
  167. And then the loading step is that process of then delivering it out, in this case, to, say, data wrapper.
  168. And so I set out to write something like that.
  169. In real life, the actual pipeline is a little more complicated, but that's really the essence of it.
  170. We're extracting from the terminal, we're transforming it into the format for data wrapper, and then we're posting to the API.
  171. The main sort of additional thing is sort of configuring each of your chart templates to behave in a certain way based on what the data source is, what type of chart you want, how you want to manage the layout, which we call the generation or the configuration of the template.
  172. Here is an example of one from real life.
  173. This is a screenshot from our code base.
  174. At the end of the day, we basically have kind of a Python framework of classes designed for each of those stages in the ETL pipeline that we then configure based on the template or chart.
  175. So this example, without belaboring it, is the unemployment chart that ran last Friday morning when the Bureau of Labor Statistics posted the new report.
  176. And again, if you were to squint and look carefully, you would see that it's really only about 20 lines of code.
  177. And most of it is really just the specific stuff related to what data file, what data series should I go get from the terminal, what should the headline be, you know, and what the date range should be and some of the specifics of how the chart ought to be, you know, collected and rendered.
  178. And that's really all it takes to do to do to do to do an individual chart.
  179. Those the system that executes that sort of pipeline process is run entirely in GitHub Actions.
  180. If you haven't checked out GitHub Actions yet, you should.
  181. This is a free task running system within the GitHub sort of repository universe where your code repository has access for free to task running jobs in data centers that will do on a schedule or based on triggers you set, run computer programming tasks.
  182. And so for each of our kind of we have our we have our templates sort of organized into groups around, say, revenue or jobs or inflation.
  183. And those groups of templates are then executed on a schedule very frequently.
  184. And when they detect that new data has been found in our source, the terminal, they make the chart and send it out to Data Wrapper.
  185. And it's that is it is the production environment.
  186. It works flawlessly.
  187. I recommend exploring things like this in your own work.
  188. Once they then deliver that to the Data Wrapper API, John's system takes over.
  189. And so this is the chart that got pushed out just one minute after the new numbers were posted to the BLS website last Friday morning.
  190. It flowed into this is the graphics pool where an editor could find, select the graphic and then embed it into a piece.
  191. It was there and ready.
  192. We're seeing these charts get delivered through the system faster than even the text, the text stories get written and edited.
  193. And they're often they're ahead of when the reporters and editors are even thinking about looking for art.
  194. So these are obviously speed wins.
  195. That chart is then put right into the story, which was on the homepage of Reuters.com and out on the wire in the next few minutes, as were several other charts in the jobs report, because we have a whole kind of package of four or five.
  196. But it's not just for speed.
  197. Indeed, you also begin to benefit from scale.
  198. So that same that same class that I showed earlier, you may have noticed that had two entries, one for the United States and the other for Canada.
  199. So that same template can be adapted to a different data series with a slightly different configuration.
  200. And boom, you now are doing two instead of one with the same amount of labor.
  201. And that can multiply to really, really large numbers in the right circumstance.
  202. So, for instance, we have a template for a bar chart of corporate earnings.
  203. So every quarter, a public corporation will report to its shareholders how much money it made in the previous period.
  204. We can make the same bar chart.
  205. We are making the same bar chart for every single publicly listed company we choose to track, which allows us to make literally thousands of these charts within seconds or minutes after the data being discovered in the system.
  206. And that's where you begin to get to scale that no human is going to be able to do on their own.
  207. So it's not just faster.
  208. It can do quite a bit more.
  209. And the result of that is that the staff on John's team is freed up to do much higher value, higher skill work, which can also have much, much higher impact as well.
  210. So in real life, you know, that once a month, every Friday morning, Allie Levine here in the New York office would have to wake up early and make that same batch of jobs reports every single time unless she missed her alarm clock, I suppose.
  211. But now the robot does.
  212. But now the robot does that for her and she gets to sleep in and do things like this beautiful Walt Disney graphic they recently published instead.
  213. And of, you know, all the people and writers that I've talked to about this project, no one has been more excited about it than Allie, who sees it as really something that allows her to work on more rewarding and challenging work.
  214. We have kind of an internal system that we've built for monitoring the output thus far, we have close to 60 templates that we've deployed,
  215. including even a few that branch out to different inputs and outputs.
  216. So for instance, we have some custom web scrapers that power charts that don't use the terminal.
  217. And we actually are publishing some static images generated not via data wrapper that get embedded into a morning email newsletter series.
  218. So the pipeline system can also kind of horizontally scale, if you will, in that way.
  219. Thus far, we've we've pushed through about 20,000 charts to data wrapper in the system.
  220. That's according to my little profile row in our our team, our team page and data wrapper.
  221. My hope is if we aren't already, we will we will have published more charts than anyone else on the platform using using this system.
  222. Beyond the actual work here privately inside of Reuters, I partnered with Sergio, the the maintainer of the Python data wrapper library.
  223. And we recently have released a new version, which I think maybe we're announcing for the first time here today now covers 100 percent of the API endpoints on the data wrapper API.
  224. So anything that's documented in the data wrapper API, we from creating folders and moving folders, organizing teams, creating maps, all of it should be possible in the system.
  225. If something isn't, tell me and I'll make sure we get it in.
  226. And Sergio has been really great to work with on that for anyone who's really into API stuff.
  227. Here's this like a quick example of of how the Python wrapper could make a really basic chart.
  228. For those familiar with the API, you'll see that that metadata variable is queuing very closely to the sort of the JSON structure that the that the data wrapper prefers to be posted in.
  229. The result of that is that it's a little it's just a little verbose and and maybe not as friendly to beginners as it could be.
  230. So, you know, one of my personal kind of dreams or goals and I would love to get feedback and collaborators on this on anyone is interesting interested would be to maybe pursue an upgrade to the data wrapper API that is more say declarative.
  231. This this is pseudo code that does not exist.
  232. But to me, this is maybe how I think the API to accept Python code where all of the attributes are sort of keyword inputs that they all would be fully documented and listed out somewhere, which I'm not sure that they are currently.
  233. And just generally trying to move towards something that makes automation easier and easier over time.
  234. So if anyone has thoughts or or interest in that, I would I would love to hear what people think.
  235. This is kind of my little, you know, attempted dreaming it.
  236. And then finally, Sergio and I recently prepared a tutorial on how any beginner could pick up Python and begin automating data wrapper charts with with zero experience.
  237. We presented that one week ago today at the the night car conference, the large data journalism conference in Baltimore.
  238. It was standing room only there was there was so much enthusiasm from the data journalists we met to do more with data wrapper and to do more with automation.
  239. I'll share the link to this in that once I finished talking here, it is free to anyone.
  240. And every step in the class is explained and all of the code is included as well.
  241. I think that's what we have.
  242. I just want to thank you all for tuning in.
  243. It's it's really appreciated, Ed, and we'd be happy to answer any and all questions.
  244. All right.
  245. Thank you so much, Ben and John.
  246. It was a fantastic speech.
  247. Very informative.
  248. Great to see behind the curtain and learn so much about the intricacies of what goes into pushing out 20,000 charts.
  249. 20,000 charts in about six months.
  250. That turnaround time that you mentioned is really remarkable and very impressive.
  251. I have a couple of questions while we wait for other questions to come through in the chat.
  252. Things that came to me while I was listening to this presentation.
  253. You mentioned a team's integration as a kind of plaxton to make sure that things were being reviewed or make sure that nothing was going wrong.
  254. So was that an idea for this project from the get go?
  255. Or was that implemented after the core pipeline was sort of built and created?
  256. Yeah, no, that was part of the original project.
  257. I mean, we needed to feed back some information to reporters about where their charts went into the graphic system.
  258. And just decided that was a really great point to go ahead and pop an image of the chart since we had that in hand already.
  259. And that gave us a chance to see what people were making very easily throughout our day.
  260. And we could pop in some comments and stuff like that.
  261. So yeah, it's actually been one of the funnest things about the whole process for me because you never know what kind of data you're going to see on any given day come through.
  262. And it's nice to talk back to the newsroom about what charts are breaking.
  263. Really true.
  264. It's also one of my favorite parts about this job is the not knowing what kind of charts are going to come through on any given day.
  265. It's very exciting and agree with you there and great foresight on that.
  266. We do have a question from Nora Gully.
  267. How do you byline automated charts?
  268. We don't put a byline in the data wrapper field.
  269. We list the source.
  270. If, you know, if the data comes from the terminal, we list LSEG as the source, of course.
  271. But, you know, no chart is published without a human editor reviewing it and what in the language of Reuters packaging it with the story.
  272. And so and so, you know, we kind of view that as the editorial oversight on the piece.
  273. If you're interested in how the quality control aspect further, I mean, we do have a committee of people that I review all the templates with before they're implemented.
  274. And so these are people who are experts in the domain that we're trying to make the chart for.
  275. And so they're involved in kind of a review process prior to that.
  276. Okay.
  277. That makes sense.
  278. Okay.
  279. I do have one other question that came to me.
  280. You mentioned that a lot of your data comes from one data source.
  281. Is that always in the same consistent format or is that ever cleaning that and preparing it for data wrapper?
  282. I can imagine that would be a sticking point.
  283. Is that always the same cleaning process?
  284. It's mostly consistent.
  285. You know, it's a truly massive database with a really wide array of data.
  286. And so it is definitely not a hundred percent consistent.
  287. And there can be, and across domains, there can be a little bit of quirks.
  288. Like for instance, commodities data is sort of, that is handled a little different from stock market data, which is handled a little different from the macroeconomic data in terms of what the columns are called or how you query it from the source API.
  289. And so that's where the Python pipeline kind of comes in.
  290. And the approach we've taken is to try to write kind of generic Python classes that handle the most common kind of variations.
  291. Hey, here's our monthly macroeconomic extractor and transformer, right?
  292. And that's going to handle most of the macroeconomic data, but then you can always kind of override or tweak, you know, the individual implementations based on what the quirks might be.
  293. Yeah, makes sense.
  294. Having that kind of room to adjust things with human eyes if needed.
  295. Fantastic.
  296. Okay.
  297. I think that we are done with any questions that might come through in the chat.
  298. I want to say thank you again so much for the time.
  299. It was an excellent speech.
  300. Great to hear about it.
  301. And now we can move on to our five.
  302. Bye.
  303. Bye.

Downloads

Slides PDF · Recording video · Extracted slide text · Timestamped transcript