Tag Archives: tips

Guest Post: DJ Patil on why Data is still sexy

By now, I’ve published several guest posts (see here , here and here) on the future of data science. Here, DJ Patil describes interesting projects in data science, giving hope to those who’ve just read the guest post on why data science is dead. A data Scientist in Residence at Greylock Partners, DJ Patil has held a variety of roles in Academia, Industry, and Government. 

Three ways data science is changing the world

Data is still “sexy”. And I don’t see any reason why that would change. Don’t believe me? Just take a look at the statistics. A quick search for data jobs on LinkedIn yields over 61,000 results and Google Trends continues to show strong growth for Data Science and Big Data. The Insight Data Science Fellows Program continues to receive over 500 applicants for each class of 30 students, with a 100% placement rate in data careers afterwards.

At the same time, we’re having some very serious debates about the acceptable use of data. For example, when is it OK to collect data or metadata (which traces the patterns of the information gathered) about the citizens of a country? And if a person is identified by mistake, whether for something as trivial as a parking offence or as serious as a no-fly list, how do you challenge an algorithm and what’s the process for fixing the error? We’re also struggling to keep some of our most sensitive data, both at a personal and government level, secure.

Data is coming under a new level of scrutiny. That’s a good thing, and I’m looking forward to seeing how the broader public debates what is acceptable. For me personally, I like to focus on some of the incredible things people are doing with data to make the world a better place.

One of my favourite groups using data for good is Crisis Text Line (CTL), which sends support text messages to teenagers in distress. The organization was started by Nancy Lublin, a Young Global Leader, and inspired by the heart-breaking responses from teenagers to texting campaigns by DoSomething.org, America’s largest non-profit for young people. As a separate entity, CTL has a single goal, which is to help teenagers through the technology medium they are most comfortable with: text messaging. Once a teenager texts in, trained counsellors reply to everything from suicide attempts, to self-harm, to bullying.

Combining this new approach with data science, CTL built their entire system from the ground up with data in mind. This includes everything from actionable dashboards to data products that help make counsellors both more efficient and effective. For example, predicting when texting volume will be high, developing queues for which counsellors are most effective, and a unique interface for counsellors to work with multiple teenagers simultaneously. These approaches have enabled them to deliver more than a million text messages in the short time the service has been running. The impact and letters from parents of the teens who have received help from the service is guaranteed to make you cry.

Another example of great use of data is DataKind. Led by Jake Porway and Craig Barowsky, it rallies data scientists from disparate places to help non-profits with some of their most pressing data challenges. They do this through a combination of “data dives” (think of these as data hackathons, open to everyone around the world) and their data corps team – in their words, “an elite group of data scientists dedicated to using data in the service of humanity.” These leading experts spend three to six months working pro bono. Their projects include getting better data about food pricing and consumption to help inform monetary policy and thwart a food crisis in Kenya, and figuring out which trees New York State should prune to stop them causing damage in a storm.

This kind of thing isn’t just restricted to the non-profit space. Companies like Jawbone (founded by another Young Global Leader, Hosain Rahman) are using data in innovative ways to help improve your health. Their data has already yielded interesting trends on sleeping patterns. And this is just the beginning, as they start to apply their insights to help personalize advice to improve your health.

Governments are getting involved, too. Code for America has had a massive impact in bringing modern technological and data science approaches to critical services provided at the city level. Each year, data specialists are paired up with a city in need. Their work really shows the merits of their approaches and the impact a little data science can have. For example, in San Francisco the team has focused on helping those needing food assistance. Their approach doesn’t just focus on bureaucratic data, but on making sure people get the help they need. At the federal level, Todd Park has been leading a similar change through the Presidential Innovation Fellows. Now on their third set of fellows, the results have been fantastic.

These data projects give me confidence that the data revolution has only just started. The people who are driving these programmes do so because they’re passionate about both data and the problems they want to solve. In the few short years these programmes and projects have been active, we’ve seen remarkable results, and I expect the impact will continue to increase. The next couple of years are going to be a great test for how comfortable we want to be with data. It is essential that we define acceptable use of data and find ways to safeguard our personal information. In doing so, we must be careful that we don’t cut off the innovation and the opportunity for data to improve lives for those who need it most.

Full disclosure, I spend as much free time I can helping DataKind, Data Insight Fellows, Code For America, DoSomething.org, and CrisisTextLine as I can — and I’m damn proud of it.

Originally Published: March 11th


With permission from DJ Patil, original post can be found here on the World Economic Forum.



Interview with BBC’s senior data architect: Jeremy Tarling

Last week I attended an interesting conference at UCL: ‘Taming the News Beast‘ with experts discussing different ways technology and the digital age can help journalists deal with the influx of data and text. You can find the liveblog of the event here.

Jeremy Tarling, senior data architect at the BBC, was there to discuss Storylines. I interviewed him afterwards on this project, coding and trainee journalists.

Listen to the highlights here, or read the full interview below.

Q: So first, can you give me a short description of what Storyline is and where it is in its production or creation?

A: The Storyline Ontology was actually a collaborative piece of work that we, the BBC took part in along with a newspaper, the Guardian; a wire service, the Press Association; a search engine, Google and ourselves as a broadcaster. The idea was that all four of those organisations represent the different types of organisations that would be interested in developing a data model for digital storytelling. The problem that the BBC was particularly interested in solving was: as we move from a world of long form articles online to something that’s more suitable for the mobile and tablet audience and for the social media using audience who refer bits of stories to each other; we wanted to find a way to make sure the bits of stories that people were sharing linked up through metadata or put into context perhaps I should say. Where the context would be the wider narrative arc so what came before, what came after, that kind of thing. And also to encourage journalists to make better use of short form content rather than always producing a long form article and then repeating most of the same things the day after.

Q: So it’s started already?

A: Yes, the project itself has been running now for about 6 months. During that time we’ve made some modifications to our content production system to allow semantic annotations with Storylines.

Q: What does that mean?

A: So for example we have a desktop content production system called CPS which is kind of like Dreamweaver, to create an article for the website, a type of tool used by journalists at the BBC. So we’ve added a new module to that tool that lets them search for current story lines or older ones. If they find any, they can attach them or tag their content with that storyline. Likewise, for topics, people, place and organisations. We’re doing the same with our video production tools as well. So we have a video production system called Jupiter- it’s based loosely on Final Cut Pro. And again, with the ability to search for current story lines and add them as metadata annotations to pieces of video.

Q: Correct me if I’m wrong, but you’ve written online that you’re not the best computer coder out there, that there might be mistakes in what you’re doing but it still works. How much coding do you know and how do you know it?

A: My background is actually not in pure computer science or even information science. My degree was in communications theory and I’ve always been a self taught programmer. My hacking language of choice is currently Ruby but it’s varied over the years. And I’ve been fortunate, in that the jobs I’ve been employed in have allowed me to develop my skills as part of my employment. But for a long time I’ve been attracted to semantic technology, ever since the early days of RDF and metadata as a way of publishing data online and linking it up. I was fortunate to be working for one of the first sites in the UK to make use of that publishing technology. It was called national curriculum online, it was funded by the department of education. It was an attempt to allow teachers to annotate teaching resources with sections of the national curriculum. Really, all I’ve done since is kind of repeat the same idea in different contexts.

Q: So it was basically learning on the job.

A: Yes, learning on the job, definitely.

Q: How important is coding for journalists, or data journalists?

A: I think it’s an emerging thing. There’s lots of talk in the industry about the importance of data journalism. And actually, if you look carefully, there isn’t a lot of good quality public data out there that’s the sort of thing that a journalist may get their teeth into and create their story from. So I think it’s not just about skill with coding or skill with statistics. There’s also the process side of data journalism which involves things like Freedom of Information requests, knowing how to get information out of public bodies, making information available in queryable form. If you can do those things then you can make your nice graphs and APIs and let people discover stories. I must admit, I’m a bit of a data journalism sceptic at the moment. In the sense that, I think, for all of the excitement about it as an emerging branch of journalism – the number of actual really good data journalism stories is still quite few. You know, you can look at the work the Guardian or Telegraph have done with Wikileaks or Snowden… is that actually data journalism or is that just a really great story with a data element to it?

Q: So how would you describe data journalism if it’s not a really great story with a data element?

A: I think data journalism is the subject of some debate. And different people use the term in different ways. I’ve heard data journalism used to describe the process of going after data, something more akin to investigative journalism. And that’s where the whole kind of FOI request and that sort of stuff comes into it. And then I think there is another school of thought that says data journalism is largely about statistics. It’s about a journalist being able to look at a large set of data and do some number crunching. Maybe some kind of statistical regression, those sort of things, to work out patterns in the data that can then be used as basis for a story.

Q: What’s your favourite current tool that you’ve used or created with regards to data to manipulate it, to play around with it? Do you have a program or an app that you would recommend?

A: No, I’m a bit of a hacker really, I don’t mean that in a bad sense, in the old school sense of the word hacker. My tools are things like the UNIX command line tools so things like scripting. I suppose, for a long time I’ve worked with Perl. And now, more recently with Ruby, but no, I don’t have any favourite kind of nice, user friendly apps.

Q: Any advice for young journalists entering the field?

A: I may be in a minority on this but my advice would be: don’t get too hung up on learning to be a programmer or developer. I mean, if you want to do that, great, that’s a career for sure, it probably pays better than journalism to be honest. But if you want to be a data journalist, the thing that drives it is good stories and an enquiring mind. And you can probably do quite well by pairing up with a statistician or a mathematician or someone that can do a little bit of coding for you, maybe graphing and those kind of things without necessarily investing a lot of time and money in training yourself up in coding courses, only to find that actually its good stories that sell. And good stories may be based on exciting data but those exciting data sets are few and far between. And probably the big guns are going to be going for them as well. I think it’s a challenging thing, I would be cautious of advising trainee journalists to kind of focus exclusively on technical skills in the interest of data journalism.




3 minutes with Jonathan Stray

Last week’s Polis journalism conference around transparency was a medley of leading men and women in the field. I managed to catch Jonathan Stray for a few minutes after his great talk with Lyra McKee and Paul Bradshaw on reducing the costs of investigations. Jonathan Stray, both journalist and computer scientist, is the founder of the Overview Project which helps journalists find stories.

Hear more on the Overview project, tips for starting journalists and what data can do here!