A tech-savvy group of analysts in New York's Office of Policy and Strategic Planning shifts through vast stores of information to help government operations run more smoothly. Michael Kirby Smith for The New York Times

 

New York - It was a case for a digital Sherlock Holmes. Last fall, the city's Department of Environmental Protection wanted to crack down on restaurants that were illegally dumping cooking oil into sewers - congealed yellow grease is responsible, the department says, for more than half of New York's clogged drains. The question was how to find the culprits.

Enter the city's Office of Policy and Strategic Planning, a geek squad of civic-minded analysts working across from City Hall in the Municipal Building. They dug up data from the Business Integrity Commission, an obscure city agency that among other tasks certifies that all local restaurants have a carting service to haul away their grease. Comparing restaurants that did not have a carter with geo-spatial data on the sewers, the team was able to hand inspectors a list of statistically likely suspects.

The result: a 95 percent success rate in tracking down the dumpers. With nothing grander than public data, the Case of the Grease-Clogged Sewers was solved.

Big Data is the tool du jour for tech-savvy companies that have realized that lurking in the vast pools of unprocessed information in their networks are solutions to some of today's most pressing and convoluted problems. A few years ago, Google, for example, took the 50 million most common keywords that Americans typed in search bars and tried to figure out, by comparing them with federal health statistics, where the H1N1 flu virus was to likely strike next.

Now the city has brought this quantitative method to the complicated machine that is New York. For the modest sum of $1 million, and at a moment when decreasing budgets have required increased efficiency, the squad has over the last three years doubled the city's success in finding stores selling bootleg cigarettes; sped the removal of trees destroyed by Hurricane Sandy; and helped steer housing inspectors - working with more than 20,000 options - directly to buildings where laws were being broken and where catastrophic fires were likeliest to occur.

"I think of us as the Get Stuff Done Folks," Michael Flowers, who oversees the group, said. "All we do is take and process massive amounts of information and use it to do things more effectively."

Before being hired in 2009 by John Feinblatt, Mayor Michael R. Bloomberg's chief policy adviser, Mr. Flowers didn't know much about computer code. But in 2005, Mr. Flowers traveled to Iraq with a team from the Justice Department to work on issues concerning mass graves and on Saddam Hussein's trial.

While serving in the Green Zone, Mr. Flowers was responsible for sending investigators to grave sites in the countryside and transporting witnesses against Mr. Hussein to his office - without getting either group blown up by roadside bombs. He learned that military officers were using predictive informational techniques to determine where and when the bombs were likely to explode.

He borrowed those techniques when he returned to New York and went to work for Mr. Feinblatt with the initial task of trying to understand what was causing mortgage fraud.

"We eventually realized there was enormous value in using all our data - together and proactively," Mr. Feinblatt said. "We'd already done the retroactive act of looking back for accountability's sake. So we tried to use the data prescriptively to figure out what might be coming next."

These days, Mr. Flowers allows the half-dozen techies working under him to ferret out answers to questions. Recently, the team was working on a project to make the city's response to natural disasters like Hurricane Sandy more robust. Catherine Kwan, 24, was correlating city information with data from utilities, like Con Edison, to put in place a system that would detect when a building's heat or lights were out.

"So what's the current ratio of Con Ed customer accounts per residential unit?" Mr. Flowers asked. (Ms. Kwan's answer: 0.88.)

With the work comes a deeper understanding of New York. The youthful quantitative analysts were surprised to learn, for instance, that it was mathematically possible to create safer streets by encouraging local businesses to keep their doors open later after dark. They also had not known that a significant percentage of complaint calls to the city's 311 municipal services line derived from certain neighborhoods in Lower Manhattan.

Mr. Flowers pointed out how complicated the city government is, mentioning the 900,000 buildings it oversees and the 10,888 metric tons of trash it picks up daily.

What the city knows about its 8 million residents is staggering. Contained in public archives is information about their boilers and their sprinkler systems, the state of their local taxes, the number of heart attacks and fires that occur inside their buildings and whether they have ever logged complaints about roaches or construction noise.

In all, a terabyte of raw information - enough to fill nearly 143 million printed pages - passes daily through Mr. Flowers's office. He is responsible for managing the online portal. "I think New York is the natural place for Big Data," Mr. Flowers said. "We have the right culture. We have a mayor who understands that management is measurement."

He added: "In New York, it's kind of like the triumph of the nerds."

Every day, Mr. Flowers said, there are 250,000 New York-centric posts on Twitter alone - some concerning trash complaints, others unsanitary restaurant conditions. If advertising agencies can use Twitter messages to sell products, he asked, "why can't the city use them to make you less sick?"

This makes civil libertarians uncomfortable.

Donna Lieberman, the executive director of the New York Civil Liberties Union, said there was potential for good in the city's attention to data, "but the potential for the selective use and release of data is one aspect that raises concern."

Another, at least for Mr. Flowers, is whether his squad will survive the end of Mayor Bloomberg's tenure on December 31. Working in his favor is the belief among information activists that Big Data's moment has arrived, a conviction based in part on a shift in how the younger generation sees its relationship to government.

"Young people, because of social media, have always felt they've had a voice," said Jennifer Pahlka, the executive director of Code for America, a volunteer group that helps governments write code for public projects. "They're coming from the assumption that government is a hackable system - an operating system that can be optimized. It's in their DNA, and they just go and do it."

The New York Times