Data rich, analysis poor

Oakland has a new School Superintendent, I like him, partly because of the following statement he dropped at a meeting of the Youth Ventures Joint Powers Authority recently- all the city and county heavy-hitters were there, discussing the possibility of hiring an out-of-state firm to do a data report on Oakland. There was much debate about the need to do this, the need for non-local data folks, the quality of local data, but Anton Wilson wonderfully cut through that: “Since I’ve come to Oakland I’ve seen a huge stream of data come across my desk. We don’t need more data, we’re not data poor- we’re analysis poor.”

I could have high five him for that statement. But that would have been awkward from across the room.

His point is one that I’ve harped on about for some time in Oaktown. We have troves of data, but barely a person doing thoughtful analysis of it to inform decision making, policy, evaluation (with the exception of some bigger programs that do get evaluated heavily).  A similar incident highlights this even more starkly. In front of the County board of supervisors, a department chief was utterly stumped when asked a seemingly simple, core metric about their department, after an injection of $75M dollars in the past couple of years for a new program.  I know that agency has tons of data, but I’m also aware of failed efforts to replace huge parts of their technology base and a stagnating effort to build a data team, so while I share some of the pain, ultimately it’s up to all senior leaders to take seriously and invest in people and systems to help make modern government agencies data smart if not fully data driven.

Part of our current problem is that the understanding of technology and data is very poor at the executive level and this often results in unwise mashing of technology and data folks with little thought to those being the right people with the right skills to actually understand your operations. I’ve talked often of the need to integrate data analysts and researchers into regular agency strategy and planning to they can respond as needs arise, but this is also a higher level problem- started when those responsible for departments do not themselves have enough data savvy or technology awareness to make good initial decisions.

If you’re one of the data geeks or tech folks in government, a good way for you to both increase your value and to help grow your organization is to add a layer of analysis or context when asked for simple data products. Instead of just giving the numbers of what you’re asked about, give some context to how that has changed, ways that measuring that thing have changed, gaps in your data that make that data fuzzy, or even better, ask those annoying questions like “What is this being used for? What decisions are you trying to make? Can I help you when it comes to digesting this information at a planning meeting?” You’ll be stunned at the number of exec level meetings with people saying ‘I don’t really know what these data mean” or “I wish we knew some context around these data”, but never bother to pass those issues down to you. Suggest you can both produce better products and also help with analysis if you are part of the process.

For leaders, humility and awareness of how much data and tech really drives the world is a powerful starting point. Look at what other progressive agencies are doing with performance management, accountability and data driven initiatives. Copy them. And perhaps most important, find a local ally who does know data driven strategies and technology management in their sleep and have them help you make better decisions. One last clue- buying business analytics software won’t help you, training your staff properly and building your capacity by hiring data and tech savvy staff will!

Are your data too slow?

Not everything can be Big Data. Not everything should be either. But some data do need a kick in the pants, so to speak. Are the data you produce or use real time, coming down the pipe as a feed everyday, or are you stuck with years old data for your planning and analysis purposes? If you are in the latter, don’t feel bad — you’re not alone.

For those tracking Ebola outbreaks in West Africa, the stream of data is steady but not real time, yet decisions that impact people’s lives are being made every day about resourcing and responding to this crisis.  In the USA there are similarly important data needed — many infections or diseases are notifiable — requiring direct notification of the Centers for Disease Control and Prevention. However regular hospital visits, treatments and surgeries go through a very big, very slow pipeline from local clinics and hospitals up to the state agency level and after processing, refining and some magical treatment, these data flow back to local public health and research agencies some years later. Traditionally this timeline was “all we could do” because of technology limitations and other reasons, but as we rely more and more on access to near real-time data for so many decisions, health data often stands out as a slouch in the race for data driven decisions.

In a different vein, campaign finance data for political donations is sometimes surprisingly fast. In California all donations to campaigns require the filing of a Form 460 declaring who gave the funds, their employer and zipcode. Campaigns are supposed to file these promptly, but this does not always happen until certain filing deadlines. Nevertheless, these data contain valuable insights for voters and for campaigns alike. These data get submitted as a flow, but they then end up in a complex format not accessible to average people — until someone changes that. A volunteer team at OpenOakland created a very powerful automation process that takes these data and reformats them in a way that makes them accessible and understandable to everyone at http://opendisclosure.io. Yet even this system of automated data processing and visualization suffers from a lack of perfectly updated data on a daily basis- the numbers shown each day only reflect the data filed to date, so big donations or changes in patterns do not show up until those are filed — often at a somewhat arbitrary deadline.

Unfortunately not all data are filed frequently and do not come with an easy to use API connection to allow developers and researchers to connect to them directly. Take crime data. Very important information with a high demand for all sorts of decisions at local levels. Your police force may publish good crime data each day or maybe just each month which is useful for real estate people and maybe good for analysts and crime investigations, but how do we know if our local efforts have successfully impacted crime? We go to national data. The Federal Bureau of Investigations (FBI) collects data from most law enforcement agencies in the country and publishes it at as the Uniform Crime Reports (UCR). Unfortunately, these data are published years after the fact. There is a convoluted process for local agencies to format and filter their reports, but then these data take years to get published.

We recently created a violent crime fact sheet using the latest (and recently published) available UCR data — for 2012. This lag in data means that county supervisors and other officials are trying to evaluate the impact of crime prevention efforts but can’t even compare their outcomes with other cities due to the lag in this data – we have to wait for two more years to see if these data indicators  changed in other comparable cities, or if our interventions did have a measurable impact.  This sort of time lag means that no local officials have good comparable data in a reasonable time frame- a poor system for modern policy makers to rely on. The FBI is working to slowly implement a newer system, but it is not clear that the lag will improve.

Every agency responsible for collecting data for operational purposes MUST start thinking about how it can make these data safely available to decision makers and to the public on an expedited process.  The technology is now very accessible to support this, and if necessary we should be considering bifurcated approaches — the old, slow feed to state and federal agencies and a new, agile feed for local use. Privacy standards and quality are simply things that guide how we can do this, they are not actual barriers unless we choose to let them be.

Government is a business, albeit one with a monopoly on services it provides — and it’s not cool for government to be making decisions using years old data when the private sector is increasingly data driven and real time. We can do this!

* First published over at Govloop

Concerned Internet Citizens of California

Net Neutrality is a big deal. My opinion and as of today the opinion of our President. The FCC is considering a rule to allow internet providers to charge premo rates to big companies to give them better speed to deliver their content to you the consumer, sounds like a reasonable idea at first? The problem is the internet was create as an open, even, fair system and was engineered to always allow for fair treatment of anyone’s content- the problem is that when Verizon, Comcast etc charge Netflix big dollars for faster pipes, they can also refuse to do so and then favor their own network content- no longer a level playing field. For small businesses this means they will no longer be able to compete in the same way- startups like, say, Facebook several years back could not afford to pay for this premium delivery, so they get wiped out- bad for innovation, bad for consumers.

That’s a short a bad summary, anyway, there’s a great engagement and democracy side to this- the FCC opened up for comments and in new open government fashion then published all the >1 million comments in raw open data for free download, yay! The nice folks at Smarter Chicago beat me to processing the data, and so you also shouldn’t mess with that, just grab the data in a nice easy format. I wanted to see how active and how vocal different communities in California were about this issue- were big cities the source of the complaints? Were small, isolated towns aware of this issue and vocal?

//open-oakland.cartodb.com/viz/73a0b130-1dac-11e4-b79c-0edbca4b5057/embed_map?title=true&description=true&search=true&shareable=true&cartodb_logo=true&layer_selector=false&legends=true&scrollwheel=true&fullscreen=true&sublayer_options=1&sql=&sw_lat=37.21064411993447&sw_lon=-125.30731201171875&ne_lat=40.641051496100395&ne_lon=-115.90301513671875

I grabbed the processed data, aggregated by City names, cleared out some junk data, combined it with Census populations and locations (I forgot how painful it is to get basic Census data these days) and calculated a simple rate- for every 100,000 people in a city/town, how many comments got submitted- neither for or against, but just how active and engaged are people in California? There are a bunch of small towns left off as their rates are not reliable. Take a look at your region, are you surprised how high or low your rate is?

I was somewhat surprised to see a few rural towns topping the commenter lists- Nevada City (oops, maybe this is my family complaining?), has the highest rate followed by Sebastapol – NorCal represent… San Francisco, the tech darling is down at 44 with almost 7,000 comments but a rate of only 782. Oaktown is less activist full than normal at #80 with a rate of 536 and dearest Silicon Valley/Palo Alto is a shameful 63rd at a rate of 665- tech city needs some more concerned residents?

The data with city by city stats are below.

Already Beyond the Data Portal

I was inspired by a recent piece by the wonderful @jedsundwall on his gov3.0 blog about the need to be going beyond data portals (much like a recent book I contributed too focuses on BeyondTransparency, shameless plug yes).

Jed totally hits it with this assessment of a growing attitude in local government towards just getting the data out:

It’s time to acknowledge that data is not made useful simply by making it available online. As we work to make data open and available, we also need to train people who can help make it accessible and useful.

In cities locally and globally the concept of open data is being pitched by vendors as a simple, turnkey thing they purchase and simply check it off their list of good government tasks.  Not enough cities have realized that this huge data resource is an amazingly underutilized and under-leveraged resource for them. In Oakland, so much of the data being published leaves much to be desired and leads to dozens of new questions about the source, quality, meaning and completeness of these data, but the city isn’t really embracing this as a way to engage the community and to see these data reach more of their potential.

Jed goes on to suggest an alternative reality where data support exists side by side with the data portals:

You’re doing your research, but you’ve heard of the San Diego Regional Data Library. You go to its website and see that you can email, call, or chat online with a data librarian who can help you find the information you need. You call the library and speak with a librarian who tells you that the data you need is provided by the county rather than the city. You also learn about datasets available from California’s Department of Transportation, a non-profit called BikeSD, Data.gov and some other data from the city that hasn’t been opened up yet.

This is where my two worlds collide. The #opendata & #opengov world is leading and pushing from a certain position, mostly not connected to the existing community research, indicator and data world and the community indicators world has been slow in embracing this brave new world of easy access to data.  We need to get along, to understand each others positions and intentions and we can really make this #datadriven world matter for our communities.

The concept of a data library is very similar to what groups like Urban Strategies Council have been doing for 15 years with our InfoAlamedaCounty.org project.  For a long time we’ve seen the need to provide communities with reliable research and data to drive action and we’ve struggled to get access to data for this entire time. 

We formed the National Neighborhood Indicators Partnership in 1992 with the Urban Institute to help build the field nationally to help empower more communities in just this way- we have a mandate to publish data and make usable, actionable information to communities equally. Our partner organizations in 37 cities have local knowledge and regional knowledge, expertise in community development, reentry, public safety, economic development, education and health, so we’re able to not just provide raw and improved data, we’re able to be an active, responsive voice in or communities to make more data actionable.

Many NNIP partners are starting to embrace the open data world and this is a powerful recipe for a data driven future that is focused on equity in our cities- most NNIP partners have a social mission as opposed to just doing data in a cold, calculated way.  But the unfortunate truth is that as our cities are becoming more data rich, many NNIP partners are facing declining funding to help support community uses of data.  It would be a mistake for funders to largely lose interest in community data intermediaries (not a sexy concept) in the excitement over open data, because none of these data become actionable and meaningful without serious support, engagement and use.

The data library is a great concept, and our experience in Oakland and many other cities says there’s huge need and value for such an entity.  Our cities can themselves play some part by being more engaged through their open data resources, but that’s never going to be enough, just like Chicago has fantastic staff who engage, there’s still a role for the Smart Chicago Collaborative effort to bring that power out to communities across the city.

More data, more engagement, more power to the people?

How to (not) support new business in Oakland

There’s a lot of excitement in Oaktown lately, lots of amazing national press coverage about how amazing our town is, how great and diverse the food and culture is, the art and music scene and the bustling new venues to enjoy. People in city hall and various chambers are talking about new business opportunities here and are optimistic about the chances for a real retail and commerce boost to this wonderful but struggling city. I’ve been chatting with people about the ways to enable and encourage new business start-ups, in both the tech and regular retail/commerce world, lots of folks are positive about spaces being leased and used from Jack London to Uptown. Then I ran across a hackathon (that is sadly fully booked, say what?) with a focus on building opportunities for commonly disenfranchised and discouraged communities to help establish new businesses in their communities, very cool TED. And I started thinking about how a new entrepreneur goes about finding the space for their new business in San Francisco, and conversely how they would do the same in Oakland. Profound differences.

In SF you quickly stumble upon the official SF Prospector, an online tool that is old and clunky from a UI point of view but never the less allowed me to pick some square footage requirements, select an area of interest, a leasing rate and some other sensible variables and then to automate the process of finding possibly suitable sites. For my theoretical new photography studio I found a good location, that happened (by pure luck I’m sure) to be on a block with several other photographic studios close by, so fortunate locality with related traffic in the area, awesome.

But if I wanted my new site to be in sunny Oakland, to establish my new business in this side of the bay? Good luck. Leg work, connections, various real estate and small business sites with no real info and no leads. Nada from the city or the county. So we have a new dream team of the city administrator Deanna Santana, well reputed deputies in Scott Johnson and Fred Blackwell and zero ability for new business owners or expanding ones to find possible new locations quickly and online and at no cost. I was really stunned, nothing official, nothing to make the path smooth to help business locate in our town. WTF.

We recently soft launched www.infoalamedacounty.org as a web mapping and data viz tool to allow people to combine public data from multiple city and county agencies, all in one place, to promote data driven decision making and real, informed planning decisions. We aimed at our main partners in the CBO community, organizers and policy makers needing better access to data. But it may be that we need to load in all our countywide property data, connect up the foreclosure filings (which are privately held and sold by the way, or scraped…) and wait till the city releases all the current business permit data and vacancy info so we can just build this ourselves. The technology is no longer a real barrier, building interactive, responsive, usable tools to support our community should be a no-brainer for our governments, but it seems very slow in coming in the east bay. Technology is an enabler, but only if you choose to use it that way.

PS If you do know of a web based tool that does help businesses plan and launch in Oakland PLEASE let me know, I’d be happy to eat my words and let people know that it is actually possible!

OpenData v Absentee Councilors

After all the excitement (mostly on my part) of finally seeing an opendata resolution making its way through city hall, just like that, pow, it’s off again. At a Finance Committee Meeting, two city councilors (Brooks, and the committee Chair, De La Fuente) didn’t show, rendering the meeting null, no quorum, hence no vote to support an opendata resolution for Oakland, hence no change happening yet, no better engagement with the tech community, no new start-up possibilities encouraged just yet…

We are on the agenda now for the next month… sure this can wait, but every extra month it takes is an extra month where our city isn’t improving, isn’t getting better PR, isn’t enabling innovative new solutions that government data can help unlock. And that’s a shame.

An Open Letter to Oakland: Innovate or stagnate.

Today in SF one of my favorite organizations had a big launch event with Mayor Ed Lee. I’m incredibly proud to have been a program mentor for Code for America in it’s inaugural year and have gained so much from my involvement with and support of CfA this past year. I swear that at every single tech, geospatial, gov and urbanist event I’ve attended in 2011 I’ve run into CfA fellows and have had great conversations and felt a huge kinship with the amazing people who have volunteered their year in service of our country.

Today the CfA CEO Jen Pahlka and the Mayor of SF announced the new accelerator startup that that will partner with cities to help streamline city processes and apply the innovation and energy of the startup tech world to the challenges of city government in SF and other cities. Also announced was the move of Jay Nath from the SF Tech Dept to the Mayor’s new Chief Innovation Officer position, congrats Jay and very impressive decision by a proactive Mayor Lee!

What does this usual tech innovation and government transformation mean to us in the East Bay? Firstly we need our city leaders and officials to recognize and support the great wealth of talent we have in the technology sector in Oakland. San Francisco is seen as the hub of tech innovation, but the reality is that Oakland has this facet also- Pandora is my personal fav of the Oakland tech startups to make it, and is a great employer in our city. Given the turnout at the first Code for Oakland event we helped run last year and the recent OpenData Day Hackathon we ran, it’s very clear to me how many interested, talented, creative technology developers we have in our town, and the reality that too many of our leaders do not appreciate is that these professionals want to support, improve, grow and celebrate our city.

Secondly we need to encourage a culture of innovation and creativity in city government, especially in the realm of technology and community engagement. I’ve had a chance to work with many city staff across different departments and can draw out a long list of problems, failures and flaws that other cities in the CfA program have also identified as weak points and have now developed open source solutions to fix these weaknesses. Currently the city is leaderless in this space and our county is not far ahead of the city. There is no support of the local tech community from city hall, no spirit of entrepreneurship emanating from the city hall, no  effort to be a platform for civic innovation and very little real engagement with this hugely talented pool of local software engineers who have a habit of finding incredible solutions to city tech issues.

This frustrates the crap outta me. As a city we have all the ducks lined up, all we need is a city structure that supports and leverages the opportunities and tools already built. Our government can and should be a platform for civic innovation and new tech startups. But what needs to happen first is that the city shows some intentional leadership on this. Like San Francisco we should create a role for an Innovation Director for the city, either in the Tech Dept or in the Mayor’s office. We should also consider the need to have the ITD director as a cabinet level position- technology is not just a bunch of back room nerds doing desktop support and the person responsible for all the city tech infrastructure should not be merely a director level with no strategic input at the Mayor’s table. This person would be empowered to motivate and mobilize our great tech community to help build new solutions for our city and to help adapt many of the tools built through CfA and in the Civic Commons to collaboratively improve our city technology solutions. Many of our tech problems have been solved in other cities and all we need to to is pick from existing open sourced applications and implement them in our town.

From my work here are a few quick areas that I’ve seen solutions for either out of CfA or in the Civic Commons:

  • Contracting processes: currently a small business contract with the city for perhaps a few thousand dollars requires the business to complete approximately 12 different documents, from word docs to locked PDFs, so they must print them all and fill them out by hand, and then submit copies. I can only imagine the city process for recording and managing these various forms when they are received. Take a look at the SmartPDF work in SF for a powerful solution, or just make the effort to combine all these forms into a single, fill-able PDF at the very least, and one day perhaps implement web based forms?
  • Adopt an Open311 system for calls for service. This platform, developed in SF and DC is an open source 311 system that has open connectors and a new public dashboard feature developed by CFA. Very powerful and no proprietary software required.
  • Work with the county to build a unified property addressing system.
  • Implement Classtalk.org across the OUSD and help our teachers keep in touch with their students via SMS – perfect for a community with low internet access at home!
  • Implement ChangeByUs, a great new tool for community engagement and collaboration.
  • Implement an OpenData policy and work with our tech community to build an OpenData portal for our city. Free up valuable city data to encourage innovation, engagement and new startups! We’re doing this anyway, but it should be supported by our city!

One glimmer of hope is the recent decision of the city Public Works agency to adopt a tool called SeeClickFix. This is the first tech move that indicates a move to more open, innovative software selection for Oakland and I’m excited about this move. We’ve already had access to this app to a degree but this new step means the city will be integrating this web based citizen reporting tool with it’s newish CityWorks platform. Finally a set of tools that have open interfaces and allow the city to connect to other systems as they are implemented. This is great, a good, forward thinking decision and we need much more of it! So congratulations for this. Seriously.

I think our city has enormous potential to become a leader in this field, to be seen as a true innovator and a city of real collaborative efforts to solve our common problems. But our leaders need to step up and be real leaders in order to see this occur. Otherwise it will happen slowly and stubbornly on it’s own, and the city will not be recognized through this but will instead grow it’s reputation as an immovable, clunky, closed bureaucracy that did nothing to leverage the immense talent and interest of it’s residents.

I love my town and I’m excited to work for an organization who supports civic technology innovation. If our city steps up we’ll support their efforts aggressively. Our town needs some positive PR, and this one is all ready to go!
peace
Spike
Read more:

the Accelerator event

To hear about the accelerator as it ramps up