Tag Archives: Data

Guest Post: DJ Patil on why Data is still sexy

By now, I’ve published several guest posts (see here , here and here) on the future of data science. Here, DJ Patil describes interesting projects in data science, giving hope to those who’ve just read the guest post on why data science is dead. A data Scientist in Residence at Greylock Partners, DJ Patil has held a variety of roles in Academia, Industry, and Government. 

Three ways data science is changing the world

Data is still “sexy”. And I don’t see any reason why that would change. Don’t believe me? Just take a look at the statistics. A quick search for data jobs on LinkedIn yields over 61,000 results and Google Trends continues to show strong growth for Data Science and Big Data. The Insight Data Science Fellows Program continues to receive over 500 applicants for each class of 30 students, with a 100% placement rate in data careers afterwards.

At the same time, we’re having some very serious debates about the acceptable use of data. For example, when is it OK to collect data or metadata (which traces the patterns of the information gathered) about the citizens of a country? And if a person is identified by mistake, whether for something as trivial as a parking offence or as serious as a no-fly list, how do you challenge an algorithm and what’s the process for fixing the error? We’re also struggling to keep some of our most sensitive data, both at a personal and government level, secure.

Data is coming under a new level of scrutiny. That’s a good thing, and I’m looking forward to seeing how the broader public debates what is acceptable. For me personally, I like to focus on some of the incredible things people are doing with data to make the world a better place.

One of my favourite groups using data for good is Crisis Text Line (CTL), which sends support text messages to teenagers in distress. The organization was started by Nancy Lublin, a Young Global Leader, and inspired by the heart-breaking responses from teenagers to texting campaigns by DoSomething.org, America’s largest non-profit for young people. As a separate entity, CTL has a single goal, which is to help teenagers through the technology medium they are most comfortable with: text messaging. Once a teenager texts in, trained counsellors reply to everything from suicide attempts, to self-harm, to bullying.

Combining this new approach with data science, CTL built their entire system from the ground up with data in mind. This includes everything from actionable dashboards to data products that help make counsellors both more efficient and effective. For example, predicting when texting volume will be high, developing queues for which counsellors are most effective, and a unique interface for counsellors to work with multiple teenagers simultaneously. These approaches have enabled them to deliver more than a million text messages in the short time the service has been running. The impact and letters from parents of the teens who have received help from the service is guaranteed to make you cry.

Another example of great use of data is DataKind. Led by Jake Porway and Craig Barowsky, it rallies data scientists from disparate places to help non-profits with some of their most pressing data challenges. They do this through a combination of “data dives” (think of these as data hackathons, open to everyone around the world) and their data corps team – in their words, “an elite group of data scientists dedicated to using data in the service of humanity.” These leading experts spend three to six months working pro bono. Their projects include getting better data about food pricing and consumption to help inform monetary policy and thwart a food crisis in Kenya, and figuring out which trees New York State should prune to stop them causing damage in a storm.

This kind of thing isn’t just restricted to the non-profit space. Companies like Jawbone (founded by another Young Global Leader, Hosain Rahman) are using data in innovative ways to help improve your health. Their data has already yielded interesting trends on sleeping patterns. And this is just the beginning, as they start to apply their insights to help personalize advice to improve your health.

Governments are getting involved, too. Code for America has had a massive impact in bringing modern technological and data science approaches to critical services provided at the city level. Each year, data specialists are paired up with a city in need. Their work really shows the merits of their approaches and the impact a little data science can have. For example, in San Francisco the team has focused on helping those needing food assistance. Their approach doesn’t just focus on bureaucratic data, but on making sure people get the help they need. At the federal level, Todd Park has been leading a similar change through the Presidential Innovation Fellows. Now on their third set of fellows, the results have been fantastic.

These data projects give me confidence that the data revolution has only just started. The people who are driving these programmes do so because they’re passionate about both data and the problems they want to solve. In the few short years these programmes and projects have been active, we’ve seen remarkable results, and I expect the impact will continue to increase. The next couple of years are going to be a great test for how comfortable we want to be with data. It is essential that we define acceptable use of data and find ways to safeguard our personal information. In doing so, we must be careful that we don’t cut off the innovation and the opportunity for data to improve lives for those who need it most.

Full disclosure, I spend as much free time I can helping DataKind, Data Insight Fellows, Code For America, DoSomething.org, and CrisisTextLine as I can — and I’m damn proud of it.

Originally Published: March 11th


With permission from DJ Patil, original post can be found here on the World Economic Forum.



Guest Post: Ryan Swanstrom on Stats and Data

I’ve already posted two arguments on Data Science and whether it’s worth going into (see here and here). Here, Ryan Swanstrom adds his two cents: the difference between what a statistician and what a data scientist does.

Data Science is more than just Statistics

I occasionally get comments and emails similar to the following question:

Should I attend a graduate program in data science or statistics?

I believe there is some concern about the buzzword data science. People are unsure about getting a degree in a buzzword. I understand that. However, whether the term data science lasts or not, the techniques in data science are not going away.

Anyhow, this post is not intended to argue the merits of the term data science. This post is about the comparison of statistics to data science. They are not the same thing. The approach to problems is different from the very beginning.


This is a common approach to a statistics problem. A problem is identified. Then a hypothesis is generated. In order to test that hypothesis, data needs to be collected via a very structured and well-defined experiment. The experiment is run and the hypothesis is validated or invalidated.

Data Science

On the other hand, the data science approach is slightly different. All of this data has already been collected or is currently being collected, what can be predicted from that data? How can existing data be used to help sell products, increase engagement, reach more people, etc.


Overall, statistics is more concerned with how the data is collected and why the outcomes happen. Data science is less concerned about collecting data (because it usually already exists) and more concerned about what the outcome is? Data science wants to predict that outcome.

Thus, if you just want to do statistics, join a statistics graduate program. If you want to data science, join a data science program.


What are your thoughts? Agree/Disagree?


With permission from Ryan Swanstrom, original post can be found on his blog Data Science 101.

Guest Post: Miko Matsumura on why Data Science is DEAD

This guest post provides an opposite opinion to John Foreman’s (See previous post). Miko Matsumura is a VP at Hazelcast who does not have a high opinion of data scientists in general, he writes: “You [data scientist] will be replaced by a placid and friendly automaton.” 

So what do you think? Are Data Scientists turning into poor substitutes for future software? Or will they constantly remain ahead of it in their ability to cohesively combine data, science, business and a host of other fields?

Data Science is Dead


Fun fact: nothing on this blackboard makes any sense.


Science creates knowledge via controlled experiments, so a data query isn’t an experiment. An experiment suggests controlled conditions; data scientists stare at data that someone else collected, which includes any and all sample biases.

Now, before you drag out the pitchforks: I’m not a query hater. You won’t see me standing outside the Oracle Open World conference with a sign that says “NO SQL” on it. Queries are fine. Smart people don’t always have the right answer, but they need to ask the right questions. Yes, building a query is like “forming a hypothesis,” but at that point we enter the realm of observational or “soft” science. Yes, by this standard, Astronomy and Social Sciences are also not sciences. I have no idea what Computer Science is, but no, it’s not a science either.

Oh what’s that? Your kind of “Data Science” includes things such as A|B Testing, and your “experiments” actually involve executing designs that affect the world? Allow me to retort: that’s not Data Science, that’s actually doing a job. You might have a job title like Product Management or Marketing. But if your job title is “Data Scientist,” you are effectively removing yourself from the actual creation of data.

I do sympathize. I appreciate that it’s no longer sexy to be a Database Administrator, and I guess the term “Business Analyst” is a bit too 1980’s. Slapping “Data Warehousing” on a resume is probably not going to land you a job, and it’s way down there with “Systems Analyst” on the cool-factor scale. If you’re going to make up a cool-sounding job title for yourself, “Data Scientist” seems to fit the bill. You can go buy a lab coat from a medical-supply surplus store and maybe some thick glasses from a costume shop. And it works! When you put “Data Scientist” on your LinkedIn profile, recruiters perk up, don’t they? Go to the Strata conference and look on the jobs board—every company wants to hire Data Scientists.

OK, so we want to be “Data Scientists” when we grow up, right? Wrong. Not only is Data Science not a science, it’s not even a good job prospect. In the immortal words of Admiral Akbar: “It’s a trap.”

These companies expect data scientists to (from a real job posting): “develop and investigate hypotheses, structure experiments, and build mathematical models to identify… optimization points.” Those scientists will help build “a unique technology platform dedicated to… operation and real-time optimization.”

Well, that sounds like a reasonable—albeit buzzword-filled—job description, no? There is going to be a ton of data in the future, certainly. And interpreting that data will determine the fate of many a business empire. And those empires will need people who can formulate key questions, in order to help surface the insights needed to manage the daily chaos. Unfortunately, the winners who will be doing this kind of work will have job titles like CEO or CMO or Founder, not “Data Scientist.” Mark my words, after the “Big Data” buzz cools a bit it will be clear to everyone that “Data Science” is dead and the job function of “Data Scientist” will have jumped the shark.

Yes, more and more companies are hoarding every single piece of data that flows through their infrastructure. As Google Chairman Eric Schmidt pointed out, we create more data in a single day today than all the data in human history prior to 2013.

Unfortunately, unless this is structured data, you will be subjected to the data equivalent of dumpster diving. But surfacing insight from a rotting pile of enterprise data is a ghastly process—at best. Sure, you might find the data equivalent of a flat-screen television, but you’ll need to clean off the rotting banana peels. If you’re lucky you can take it home, and oh man, it works! Despite that unappetizing prospect, companies continue to burn millions of dollars to collect and gamely pick through the data under respective roofs. What’s the time-to-value of the average “Big Data” project? How about “Never”?

If the data does happen to be structured data, you will probably be given a job title like Database Administrator, or Data Warehouse Analyst.

When it comes to sorting data, true salvation may lie in automation and other next-generation processes, such as machine learning and evolutionary algorithms; converging transactional and analytic systems also looks promising, because those methods deliver real-time analytic insight while it’s still actionable (the longer data sits in your store, the less interesting it becomes). These systems will require a lot of new architecture, but they will eventually produce actionable results—you can’t say the same of “data dumpster diving.” That doesn’t give “Data Scientists” a lot of job security: like many industries, you will be replaced by a placid and friendly automaton.

So go ahead: put “Data Scientist” on your resume. It may get you additional calls from recruiters, and maybe even a spiffy new job, where you’ll be the King or Queen of a rotting whale-carcass of data. And when you talk to Master Data Management and Data Integration vendors about ways to, er, dispose of that corpse, you’ll realize that the “Big Data” vendors have filled your executives’ heads with sky-high expectations (and filled their inboxes with invoices worth significant amounts of money). Don’t be the data scientist tasked with the crime-scene cleanup of most companies’ “Big Data”—be the developer, programmer, or entrepreneur who can think, code, and create the future.


With permission from Miko Matsumura, original post can be accessed here on Dice.

Interview with BBC’s senior data architect: Jeremy Tarling

Last week I attended an interesting conference at UCL: ‘Taming the News Beast‘ with experts discussing different ways technology and the digital age can help journalists deal with the influx of data and text. You can find the liveblog of the event here.

Jeremy Tarling, senior data architect at the BBC, was there to discuss Storylines. I interviewed him afterwards on this project, coding and trainee journalists.

Listen to the highlights here, or read the full interview below.

Q: So first, can you give me a short description of what Storyline is and where it is in its production or creation?

A: The Storyline Ontology was actually a collaborative piece of work that we, the BBC took part in along with a newspaper, the Guardian; a wire service, the Press Association; a search engine, Google and ourselves as a broadcaster. The idea was that all four of those organisations represent the different types of organisations that would be interested in developing a data model for digital storytelling. The problem that the BBC was particularly interested in solving was: as we move from a world of long form articles online to something that’s more suitable for the mobile and tablet audience and for the social media using audience who refer bits of stories to each other; we wanted to find a way to make sure the bits of stories that people were sharing linked up through metadata or put into context perhaps I should say. Where the context would be the wider narrative arc so what came before, what came after, that kind of thing. And also to encourage journalists to make better use of short form content rather than always producing a long form article and then repeating most of the same things the day after.

Q: So it’s started already?

A: Yes, the project itself has been running now for about 6 months. During that time we’ve made some modifications to our content production system to allow semantic annotations with Storylines.

Q: What does that mean?

A: So for example we have a desktop content production system called CPS which is kind of like Dreamweaver, to create an article for the website, a type of tool used by journalists at the BBC. So we’ve added a new module to that tool that lets them search for current story lines or older ones. If they find any, they can attach them or tag their content with that storyline. Likewise, for topics, people, place and organisations. We’re doing the same with our video production tools as well. So we have a video production system called Jupiter- it’s based loosely on Final Cut Pro. And again, with the ability to search for current story lines and add them as metadata annotations to pieces of video.

Q: Correct me if I’m wrong, but you’ve written online that you’re not the best computer coder out there, that there might be mistakes in what you’re doing but it still works. How much coding do you know and how do you know it?

A: My background is actually not in pure computer science or even information science. My degree was in communications theory and I’ve always been a self taught programmer. My hacking language of choice is currently Ruby but it’s varied over the years. And I’ve been fortunate, in that the jobs I’ve been employed in have allowed me to develop my skills as part of my employment. But for a long time I’ve been attracted to semantic technology, ever since the early days of RDF and metadata as a way of publishing data online and linking it up. I was fortunate to be working for one of the first sites in the UK to make use of that publishing technology. It was called national curriculum online, it was funded by the department of education. It was an attempt to allow teachers to annotate teaching resources with sections of the national curriculum. Really, all I’ve done since is kind of repeat the same idea in different contexts.

Q: So it was basically learning on the job.

A: Yes, learning on the job, definitely.

Q: How important is coding for journalists, or data journalists?

A: I think it’s an emerging thing. There’s lots of talk in the industry about the importance of data journalism. And actually, if you look carefully, there isn’t a lot of good quality public data out there that’s the sort of thing that a journalist may get their teeth into and create their story from. So I think it’s not just about skill with coding or skill with statistics. There’s also the process side of data journalism which involves things like Freedom of Information requests, knowing how to get information out of public bodies, making information available in queryable form. If you can do those things then you can make your nice graphs and APIs and let people discover stories. I must admit, I’m a bit of a data journalism sceptic at the moment. In the sense that, I think, for all of the excitement about it as an emerging branch of journalism – the number of actual really good data journalism stories is still quite few. You know, you can look at the work the Guardian or Telegraph have done with Wikileaks or Snowden… is that actually data journalism or is that just a really great story with a data element to it?

Q: So how would you describe data journalism if it’s not a really great story with a data element?

A: I think data journalism is the subject of some debate. And different people use the term in different ways. I’ve heard data journalism used to describe the process of going after data, something more akin to investigative journalism. And that’s where the whole kind of FOI request and that sort of stuff comes into it. And then I think there is another school of thought that says data journalism is largely about statistics. It’s about a journalist being able to look at a large set of data and do some number crunching. Maybe some kind of statistical regression, those sort of things, to work out patterns in the data that can then be used as basis for a story.

Q: What’s your favourite current tool that you’ve used or created with regards to data to manipulate it, to play around with it? Do you have a program or an app that you would recommend?

A: No, I’m a bit of a hacker really, I don’t mean that in a bad sense, in the old school sense of the word hacker. My tools are things like the UNIX command line tools so things like scripting. I suppose, for a long time I’ve worked with Perl. And now, more recently with Ruby, but no, I don’t have any favourite kind of nice, user friendly apps.

Q: Any advice for young journalists entering the field?

A: I may be in a minority on this but my advice would be: don’t get too hung up on learning to be a programmer or developer. I mean, if you want to do that, great, that’s a career for sure, it probably pays better than journalism to be honest. But if you want to be a data journalist, the thing that drives it is good stories and an enquiring mind. And you can probably do quite well by pairing up with a statistician or a mathematician or someone that can do a little bit of coding for you, maybe graphing and those kind of things without necessarily investing a lot of time and money in training yourself up in coding courses, only to find that actually its good stories that sell. And good stories may be based on exciting data but those exciting data sets are few and far between. And probably the big guns are going to be going for them as well. I think it’s a challenging thing, I would be cautious of advising trainee journalists to kind of focus exclusively on technical skills in the interest of data journalism.




Taming the News Beast Liveblog!

The International Society for Knowledge Organisation (ISKO) is hosting Taming the News Beast at UCL.

Click here to see the liveblog of the event from 14:00 on the 1st April.

This event will discuss how to manage the information overload digital world offers. Leading practitioners will discuss the intersections with technology and news ways journalists can handle large quantities of text and data.

Speakers include Data Architect Jeremy Tarling from the BBC and Pete Sowerbutts from the Press Association.


13:00 ISKO UK AGM – for members – observers welcome
13:30 Registration for Taming the news beast
14:00 Welcome and introduction
Stella Dextre Clarke, ISKO-UK Chair
Helen Lippell, Meeting Chair
14.10 BBC News Labs – the Newsroom of Things
Matt Shearer, BBC
14.35 Storyline ontology
Jeremy Tarling, BBC
15.00 Taming the ABC News Beast – a case study
Rob Corrao, LAC Group
15.30 Refreshments
16.10 Text analytics for news and social media
Ian Roberts, University of Sheffield
16.45 Embedding semantics into content
Pete Sowerbutts, Press Association
17.15 Fishbowl: question/answer session
Audience-led discussion of issues arising from the presentations
17.45 Close of meeting
17.45 Networking, wine and nibbles in the Ramsay Lecture Theatre