“We may yet employ our vast technology to the task of obliterating ourselves and life on earth. This possibility should surprise no one: throughout history, humankind has exhibited an enthusiastic genius for establishing hells on earth that surpass the misery of those conceived by our poets, artists, and theologians. On the other hand, the ability to deliberate together may be our most powerful—yet neglected—natural resource. And in our embrace of open governance, we may discover that it is the key to civic intelligence.”

Did you think your postman/woman just delivered your mail? Wrong. They happen to be a rather awesome real-time crowd sourcing force for public data! When your mail is delivered and you turn out to not be there, and this keeps happening for a month, or for three months, your responsive public servants take note of this, it goes into a big database and gets spat out courtesy of HUD in a tract level file. It also happens to track business vacancies. Check it.

http://a.tiles.mapbox.com/v3/urbanstrategies.map-yn73atfz.html#9/37.411933708075516/-121.40486523437498

Some truthiness to know- no data are perfect. These suffer from many variations and local contexts that make them tricky at times, but they are an amazing resource, and these data get republished every quarter. I threw together the map above in a couple of hours, part of my playing/learning with MapBox, which I fricking love already. My biggest delay was generating these tiles for the web and realizing I had left a background layer turned on rendered blue which prevented the street layer from showing through. You learn that lesson once. Waiting 4 hours for a render job and getting unusable map layers is a bad experience.

Check the full screen version out here:

http://tiles.mapbox.com/urbanstrategies/map/map-yn73atfz

Nice map, but I want the data dammit! Fine, have it. Both shapefiles and clean csv files on our data portal: http://data.openoakland.org/en/dataset/vacancyratesusps

I do get that the multiple decimal places is annoying, but I’m not re-doing this again… the map hover/click functionality also gives you the business vacancy rates too, as the baseline data for how many units of residential and commercial are in each tract for context.

It’s a rare and wonderful thing to be so involved in a city like Oakland, to get to be part of so many awesome organizations, initiatives and projects and to have access to so much rich data and personal knowledge about our city. It’s also challenging to face the fact that so much of our institutional and community knowledge is locked away in the minds of our elders, our colleagues, our servers, and hence is not accessible for our community at large. Unless they schedule time to talk to said people. This creates a problem of knowledge scale and transfer, of concentrated soft power. I’m calling it soft because it doesn’t really make you a boss to know stats or history of a community, but it is still powerful to have access to so much info on your community, in the right setting.

For the past few years I’ve been more focused on publishing our work widely, not to make us look like experts, but to actually share what we do, what we learn, what we care about.  It seems there’s another level needed here too- sharing what we know, and even deeper, sharing what we share with others in intimate settings. Every question asked of you- chances are someone else wants that answer too. And they won’t ever get to ask you.  The data you issue someone in response to a request, the facts you smugly share about something you know that the asker didn’t.  If we only share things once, in non-transferable formats we are not doing all we can to democratize knowledge, to make the best use of what we know.

In the past this has meant we publish our internal project maps and reports for others to use, then we publish raw data, then we make it usable for others.  The next step may be to wiki our responses.

Yes, wiki.

An example from today. Someone mentions something on twitter about neighborhoods and their definitions in Oakland, about how they vary.  In response, to help inform this conversation about a city I love and know something about I have a couple of options. I can make a benign tweet response, I can maybe email a link or share a photo, or I can pass along information in a more permanent way, in a way that can perhaps help others.  This later option is becoming my default. Instead of a light, individual contact with an inquiry or a random interest, I can put down some of what I know, some of what I have (data, reports) and put it in context (hopefully) and make it available for everyone. It turns out that a local wiki like OaklandWiki.org is perfect for this!

It took me about 10 minutes to update a page on a particular neighborhood that was a perfect demonstration of the issue being discussed, (Lower) San Antonio. I can include some of what I know through research, through relationships, share data snippets and content relevant to the community. Fast, easy, permanent and open to anyone.  Instead of just a small channel of communication, I can share a response, a story, a factoid with anyone who bothers to google it. In a sector obsessed with Social Impact, it seems one of the highest yield things we can do is to share what we know, what we have, especially when those things are not openly available to others in our community!

This applies to all of us- Oaklandwiki.org is not just a system for “experts”, it’s open for anyone who knows something about Oakland to contribute, to share.  Just like libraries were the place we stored and made available “what we knew”, local wiki’s are increasingly becoming the place where we can all share and access “what we all know”.

Oakland has a huge network of Neighborhood Crime Prevention Councils that provide a level of community interaction and engagement around crime and local issues.  This network is supported by city staff (Neighborhood Services Coordinators- NSCs) and Problem Solving Officers (PSOs) and each has a local member as the chair. In come neighborhoods these groups operate more like planning councils on broad topics, in others you’ll find people talking about stray dogs, blight, violence and truancy.  These are mechanisms to engage residents on civic issues. they matter to a city. We have a huge list of all the different yahoo groups where they communicate if we’re interested, but how, just how do you find out what beat you are in?

The city’s current answer: use this incredibly horrible PDF document to clumsily guess. Seriously. I swear the city had something better in the past, but this is how you are meant to find and connect to your “local” group. Even ignoring its purpose, this is one of the worst maps I’ve ever seen in Oakland.

Fortunately I do believe that our interfaces to government can be beautiful, can be simple and can actually function. And I believe that civic engagement and action is critical in a city with such structural problems. So I built a better version. In less than three hours. I give you an open source tool that I originally cloned from a great developer in Chicago and remade as a tool to easily help parents find which elementary school zone they live in, where your free tax sites are in the Bay Area to get awesome free tax prep help from professional staff if you earn under about $50k.

The new web app is found here: http://spjika.github.com/FindYourBeat

It’s simple, does just one thing but hopefully does it well. It’s a small tool that you should only need to use once in your life per house you inhabit.

Knowing how to use the tools and how to find and redeploy open source code means you can produce something incredibly useful and interactive in about the same time as you would take to make a half decent static map as a PDF.  I can say this because I’ve made literally hundreds of good, static PDF maps.

There are obviously dozens of other areas where our local government interfaces are not customer friendly. And we can help to improve them.  At OpenOakland we love to hear about areas that better technology can improve government interactions, citizen engagement and efficiency. Have an idea? We’ll soon have our new project requirements criteria published so you can pitch a new tool, process or tech idea!

If you are confused by the concept of open source, it means that anything you build, you share the source information publicly, freely, for reuse. It can be software code, it can be data and the analysis method you used, it can be a policy! By sharing it, you enable others to do even more with what you made. Just like the app that OpenCityApps built and shared, the app that sparked a hundred websites…

CapGemini recently published their report on the progress and success of global open data initiatives, at a country level. It has some really insightful data and points, and isn’t a tough read at 17 pages. As my city Oakland, and our county, Alameda, launch their open data platforms I think this national level report has a lot for us to consider about local efforts.

Some broad concepts that we seek to capture:

Open Data drives growth by stimulating the creation of firms that reuse freely available government information in innovative ways.

This is the macro promise, and it’s a clear correlation.  In the San Francisco East bay it will be telling if these benefits are local, where the data live, or dispersed and hard to measure as local benefits of local efforts.

And now the test of time begins:

96% of the countries analyzed in our research shared data which is not regularly updated.

This is the ugly stuff- maintenance. It’s great to get publicity friendly wins on the board with data releases, cool new apps and tools, but in a year down the road, if those new data are the same data then we’re just made a blip on the heart rate monitor and it’s not looking like a living, breathing being anymore.

But we need to be honest in the civic innovation/hacking world too- there have been incredibly powerful and cool things built by amazing developers, especially the work of Code for America, but many of these things created in past years that have not been commercialized face an ugly reality- the task of maintenance.

Google learned some good, hard lessons with updating road data for Google maps (Apple Maps seems to be doing more of the hard and less of the good so far). I have some mixed feelings about a lot of tech I love to use- uncertainty about the ease which we can manage both granular and major changes to the underlying data used. If users add to the administrative data it’s a great collaboration, when the underlying data shifts slightly however, we have some tough work to do. Let’s not forget that. Enterprise and government level tech frequently sucks, in some part because it requires full time managing of the data. I’m hoping our more cutting edge tools stand up to these tests in the coming years.

Sustaining open

Open Data initiatives need to be driven from the top with strong political leadership.

We now have two great resources in Oakland/Alameda. City and County staff have listened to those of us who encouraged these efforts and have done good work to implement the tools and practices to support open data. What is now clearly absent from our efforts is the legislative support to truly sustain and mandate these initiatives don’t just disappear at the whim of any bureaucrat or budget adjustment. No US cities or jurisdictions I’m aware of have done it quite like we have here. These efforts have been driven by informed political leaders who have implemented laws and orders to support and sustain open data. We are now at that time in Oakland & Alameda. Our mayor and councilors, our supervisors need to now swiftly adapt legislation used in many other cities (a very easy task, heck I’ve done it for them already and given them copies) to show not only their support for this efforts, but to clarify their vision and intent in law.

I’m looking to our leaders to really drive these efforts, to provide a vision that puts them in context and laws to ensure these great efforts don’t die on the vine like too many other good efforts have.  Open government is good government, it also happens to be smart government. Data is now becoming open and public, now you can really engage your communities  with and through these data!

The original report PDF