In a world of increasingly open government data, who is going to create the services?

Brian Hoadley has a powerful go at the answer:

Those who campaign for the release of Government data seem to fall into a few major camps:

  • Those who want more access to information because it will inform their work – e.g. the press via MP Expenses
  • Rights activists who once the data is free will move onto another cause – because that’s what they do
  • Those individuals who encircle Government who continually talk about how they could produce far better ‘Services’ than Government, at a fraction of the cost and time

Better access to data for those who monitor Government and then report on its activities will have certain benefits. We can all agree that some portion of the expenses scandal was beneficial and could lead to positive change in Government spending policy. We should also acknowledge the reality – that probably 80+ percent of the scandal was merely spectacle to earn revenue for news organisations.

I will admit that the efforts of rights activists will help groups 1 and 3 above by fighting a meticulous battle to gain access to what many term as Public data in any case.

But what about those ‘Services’?

To understand the drive behind this, we need to understand that with the Government in a precarious position due to over-extension of resources during the Recession, anything that could lead to a reduction of costs will look attractive. Take, for example, the appointment of a Digital Inclusion Champion to get the remainder of the UK population online.

Why would the Government do this?

Because long-term, the consumption of digital services, that can accommodate millions in the way a physical location cannot, will result in cost savings through the reduction of said facilities and staff to run them.

My instincts on this are very much like Brian’s:  industrial strength systems require some form of industrial strength management.  But it doesn’t at all follow that nothing has changed or will change (and I don’t imagine that Brian is suggesting otherwise).  There are several important forks in the path, which separately or cumulatively could lead to our ending up in quite a range of different places.

1.  Personal and impersonal

The focus of open data is very much on impersonal and aggregate data:  postcodes, mapping, crime statistics, school performance and a whole very long list more.  What all of that data has in common is that it can all just be handed over for people to play with and build new things.  Leaving aside the surprising ability of impersonal data to become personal in slightly unexpected ways, that can all be open and straightforward.  The issues suddenly get very different once personal data comes into the picture, because the same spirit of playful openness is simply not an option.  That’s not at all to say that personal data can only be handled inside government organisations but rather, as some of the points below start to explore, that different approaches and tools are needed to building services which deliver personal outcomes.

2. Front end and back end

Front ends can and should be simple, back ends often need to be complicated.  That doesn’t make either one inherently easier; it means that they are different kinds of problems.  Achieving elegant simplicity at the front requires hard work, but very different hard work from that required to achieve robust completion at the back end.  Brian goes on to talk about FixMyStreet, the inspired genius of which is that it doesn’t make the slightest attempt to solve the back end problem, it simply presents information to local authorities to do with what they will.  The LA then needs to diagnose what kind of problem it is and whose responsibility it is to solve it, identify and allocate resources, integrate with existing plans, schedule activity, undertake task, record completion and, ideally, somewhere along the way consider whether the problem could have been avoided in the first place.  I can see no obvious sign that systems to support that set of activities are going to come from anywhere other than their current sources (which is not to say that they will continue to be designed and built in anything like the same way).

That’s not at all an argument for government doing everything. Even big, complex, sensitive systems don’t have to sit wholly within government.  A large majority of tax returns reach government as structured data without having touched a government front end, because a whole lot of third party providers have found it in their interest to create front ends.

3.  Inside and outside

The question of how to do all this still tends to be framed round the assumption that it is government which holds personal information, that it has an obligation to limit access to and use of that information and that issues such as joining up services and sharing the data necessary to do so are problems which need to be solved and which only government can solve. Shifting the primary data store away from government (and other service providers) altogether – the volunteered personal information model – is one way of reframing the question.  But even that doesn’t take away the government big systems problem:  the piece of information you chose to share will often (but certainly not always) itself need to be stored in order to provide the service, to smooth future service provision and to provide assurance that the right things have been done.

4.  Facebook and Prince 2

Choices on those first three dimensions between them open up a huge range of futures, with no reason to suppose that any one of them will become the single universal model.  But unless we do something even more radical, they all still need there to be big transactional, personal systems (though they don’t necessarily require those systems to be owned or operated by government).  Brian’s answer is simple and, I suspect, right:

So who, in reality, will create those digital services? It will be same internal teams, companies and consultancies who currently work for Government.

In practical terms, they are the only ones who have the infrastructure and capital to go through ISO accreditation, PRINCE training, supply account and project directors, planners, technical architects, UCD experts, designers, developers, testers and hosting.

The argument in the past has been that those techniques are the only reliable way of delivering systems with the scale and resilience needed.  But the critical question now is not whether big complex systems are needed, but whether there is only one way of building them.  I have written before about the Facebook example, which is one of several which challenges the idea that robust large scale systems supporting high volumes of rapidly fluctuating personal data can only be managed in one way

None of that means that we will ever stop needing fast moving and often small scale innovation.  There are far too many examples of attempts to build big things in one go where either it proved too big to succeed, or got finished only when the rest of the world had moved on to something else –  or both.  But it does make the big challenge of the next few years look more about turning ideas and prototypes into boringly robust services than about generating new ideas – which is not to say that we won’t still need new ideas, just that it’s pretty clear that they will continue to come.

That’s why I was so encouraged to see Rewired State taking a step in this direction when I wrote about it earlier this week.  They are moving in from one end of the spectrum.  Now we need to get some movement from the other end.


  1. Great comments on how to manage the floodgate of information. I strongly believe that when you combine social networking with an easy way to manipulate the data in one place, we will see groups of individuals emerge from all areas who will become the ones that people follow.

    Experts are not necessarily the ones who look for new insight from this data. But people who have borderline interests in a topic will find other related information and create new data sets of real information. That is why our work on the Appeta platform is going to change the way we look at data and who gets to play with it.

  2. Some great thoughts there – and reading between the lines of your conclusion, perhaps you have some optimism for the future?

    I’m not so sure, based on what we’ve seen so far. There is an enormous gulf still – and for the foreseeable future – between the creative ideas and talent on the outside for front-end innovation using open data and APIs, and the reality of service delivery programme management, procurement and technical vision on the inside.

    Brian’s conclusion – that the same people building services now will continue to do so, using the newly open data, strikes me as only half likely: what if the same people just keep building in the usual ways, and ignore the RDFa and CSVs being unearthed? After all, what incentives are there for them or the people commissioning them to do anything else? The incentives the other way are still relatively strong (system security, prima facie efficiency, data integrity etc)

    I’m interested in at least a couple of angles to this:

    – how can the procurement of transactional public services online be managed in ways which bake in support for outside innovation and potentially, for outside delivery lock, stock and barrel? Can we make APIs, permissive licences, modular technology choices, open source consideration etc mandatory? And can we procure things in ways which the denizens of Rewired State can help to deliver? I’m conscious there’s a risk of celebrating their talent, passion and ideas at hack days but not doing anything much which would translate that into meaningful change in how services get delivered online.

    – who owns these transactions, in terms of small-p politics? This one is perhaps the most crucial of all – if accountability and delivery are spread across multiple teams, organisations and platforms, we’re surely destined to fail to deliver programmes with the necessary customer focus and efficient project management methodologies. Can we be more radical than previous attempts have been to establish ‘customer champions’ in government that have real teeth? Of course, I’m not saying that distributed, decentralised teams can’t work – just look at most open source projects – but that in delivering transactional services, people need to acknowledge shared ownership where it exists and have some process for governing it.

  3. A very useful synthesis of the challenges ahead in using public data and semantic technologies to deliver robust and critical systems for government – managing the ‘enterprise data of government’.

    Isn’t it about moving from (or at least balancing) enterprise design with flexible progression and demonstrating the value of allowing more powerful aggregations of data?

    Some steps forward:
    – persistent and uniform URIs
    – capture provenance
    – focus first on de facto local standards to do what you need to do
    – Should build trust and encourage reuse

Comments are closed.