Wrestling with Data in the Big Apple: Strata Summit 2011

start discussion

O'Reilly Conferences

Mini Maker Faire during Strata Conference.

Hundreds of developers and data junkies descended on Times Square last Tuesday for Strata Summit, a two-day conference to grapple with the problems and possibilities of ever-growing data sets. The Bay Citizen was in attendance — here are some highlights:

Robert Kirkpatrick, director of the United Nations' Global Pulse Initiative, came to Strata calling corporations and global citizens to action.

Kirkpatrick wants business to release anonymous data that can be used to identify and predict crises like famines, droughts or economic depressions. In his own investigations into publicly available data, he found that tweets from Indonesia revealed power outages and lines of villagers waiting for fuel. Prices scraped from e-commerce sites in Latin American allowed him to track food inflation in real time. 

“People share so many things online today that we care deeply about at the UN,” Kirkpatrick said, including data that could influence policy makers' decisions, if it were readily available. But he said much of the important data is in the hands of private companies.

Kirkpatrick announced a new social network, Hunchworks, to allow experts diving into data sets to connect and discuss ideas without the formalities of academic research. See Kirkpatrick's talk here or read about his agenda here.

Continue Reading

Summer of Smart: The Hackathons Continue!

start discussion

Hackathons have been popping up throughout San Francisco this summer, conjuring images of programmers coding under fluorescent lights all weekend with the goal of building something by the time Monday rolls around. Hackers might leave to sleep overnight, but some don’t even do that, and end up coding well into the night.

I’m not sure whether they slept, but last weekend’s Summer of Smart hackers hit the keyboards and the streets, venturing into the Panhandle and even MUNI headquarters to bring their ideas to life.

“The idea is to take an idea as far as you can in 48 hours,” said Jake Levitas, research director at Gray Area Foundation for the Arts (GAFFTA), the event organizer. “I was completely blown away by the quality of the projects–I’m not just saying that.”

Summer of Smart is GAFFTA’s three-month experiment to bring different kinds of urbanists together to generate new projects and tools over three hackathons. The best projects will be presented to the mayoral candidates at the end of the summer. (The Bay Citizen is among GAFFTA’s media partners--check out my live tweeting of the event here.)

The event kicked off with keynote speakers Christine Outram of MIT’s SENSEable City Lab, which created the Copenhagen Wheel; Brandon Tinianov of green buildings advocate Serious Energy; and Morgan Fitzgibbons of the Wigg Party, a organization focused on community-building along San Francisco’s "Wiggle" bike route. Rebar Group’s Matthew Passmore--the guy behind something called a bushwaffle--ended his speech with some words of encouragement to the attendees.

Continue Reading

Bay Citizen's Bike Accident Tracker Honored by Knight Batten awards 2011!

1 Comment

The Bay Citizen's News Apps team has won a prize at the Knight Batten awards for innovation in journalism for our Bike Accident Tracker!

Today's big winner was Storify, the social-media curation tool created by Hacks/Hacker's Burt Herman and others. Our partner site Texas Tribune, NPR's Andy Carvin and his Twitter community, Bloomberg Government and West Africa Democracy Radio also received prizes.

The Bay Citizen was one of another four outlets to receive an honorable mention and a prize at a seminar at the Newseum in Washington, D.C. in September.

This year’s winners were selected from 123 entries.

"This year's innovations demonstrate how journalists keep developing clever ways to open up the process of journalism to new contributors and seekers of information," said Jose Zamora, Knight Foundation journalism program associate.

In previous years, the Knight Battens have rewarded innovative sites such as ProPublica or Ushahidi.

Read more on the Knight Batten awards for innovation in journalism.

Bay Citizen + Hacks/Hackers Code-a-thon: Hacking DataSF

start discussion

Thanks to all who came out this weekend for our first-ever hackathon! For those who missed it, our theme challenge was DataSF: find a dataset from the City of SF's online repo of public data and do something cool + useful with it.

How To Improve DataSF?

Before diving in, we had some great discussion on how to make this platform--and the open gov initiative everywhere--work better. Jay Nath (@Jay_Nath), the open-data pioneer who works at the City of SF's Department of Technology and helped spearhead the creation of DataSF, kicked things off Saturday morning with a great backgrounder on how SF got its open-data platform and what the future holds. It was interesting to learn that the city is moving DataSF over to Socrata, the cloud-based open-data platform that already stores public data for Chicago, Seattle, the federal data site Data.Gov, and recently landed its first national customer: Kenya.

Jay readily acknowledges that while launching DataSF was a good first step in the direction of opening up government in San Francisco, the platform could be way better. The biggest challenge seems to be encouraging participation from city workers: DataSF is only useful if the city's best, juiciest, most valuable numbers are constantly being uploaded to the site (and of course, some in city government stand to benefit if that never happens).

Continue Reading

Django/DocumentCloud integration? There’s an app for that!

start discussion

When it comes to scrubbing important documents for an investigative project, there’s no better tool than DocumentCloud. But when it came to managing those documents through our own Django-powered CMS, as opposed to relying on the DocumentCloud web interface each time, we were out of luck. So we built a bridge and threw the code into GitHub. Now, other newsrooms running Django can fork, clone, and contribute to this project.

WHY

 We were already using our CMS to upload important documents to AmazonS3 so they were accessible to all our reporters but those files were flat and didn't facilitate collaboration.  DocumentCloud has all the features a good newsroom needs when it comes to collectively scouring documents but it wasn't an integrated experience.  A reporter would have to jump back and forth between their newsroom’s CMS interface and the DocumentCloud web interface, writing the story in one and place and uploading and accessing their documents in another. For a reporter on deadline, the document upload could be sacrificed for time.

So we created this app to let reporters upload and access their documents directly through their newsroom’s CMS, saving them time and paving the way for a smoother method of publishing and embedding documents with stories.  Special thanks to Ben Welsh for building the python-documentcloud api wrapper which made this app dead simple to implement.

Continue Reading

Gavin Newsom, Joanna Rees, and a roomful of hackers: Summer of Smart continues

start discussion

Over the weekend, around 50 artists, city planners, journalists and developers gathered on a fifth-floor office space overlooking ramshackle Market and Sixth streets with a single goal: build an interactive project focused on art and community in 48 hours or less.

Courtesy of GAFTA

Calling All Developers, Journalists and Civic Thinkers To Our First Codeathon

1 Comment

The Bay Citizen and Hacks/Hackers are teaming up to present our first codeathon!

It's a 24-hour event focused on pulling info from DataSF.org and building cool and useful things with it. We're really excited about finding new ways to share information about our city through visualizations, storytelling and creative use of public information.

The event brings together journos and developers, designers and civic thinkers to play with data, meet talented people and a chance to publish their work at The Bay Citizen. 

The codeathon is July 16th-17th at The Bay Citizen's offices. Kickoff is at 10am and runs through 5:30pm on the 17th. Attendees are welcome to stay overnight at The Bay Citizen's offices.  

Check out our eventbrite page for details and to register.  See you there!

Census: These Are the People in Your Neighborhood

start discussion

The U.S. Census Bureau has launched an extremely useful -- and fun -- interactive tool that can help you bore in on the characteristics of an area's population with an incredible level of detail.

Just put in your address on their interactive population map and you can see -- down to the block level -- how many people the 2010 Census counted, their age, race, age, and gender. You can use the interactive map to see how many of your neighbors are tenants and how many are homeowners. 

For example, just now I learned that the my block in Southwestern San Francisco has 169 residents, including 79 Asians, 39 African Americans, and 28 whites, along with 15 Hispanics (who may be of any race).

There are 57 children living on my block and 16 senior citizens.

Forty-three of the homes are owner occupied, while ten of them are rentals. Three units were vacant at the time of the census. Two were for rent and one was for sale.

If that's not enough for you and you are more of a computer geek and are able to manipulate an extremely large .csv file, you can see everything the Census knows about the Golden State and all its communities by downloading the Bureau's Summary Data file for the state of California.