Hackathon Project: CSV 2 API Pitch (Marcus)


#1

Here’s a synopsis of my (@pacharanero) pitch, made at the first The Deal Hackathon, at Leigh Hackspace, 5th-6th March 2016.

##Problem 1

  • there is a lot of open data about Wigan Borough, but often it is not in a suitable format for consuming by applications (eg mobile apps, web apps, IoT etc) - in order to be in an easily consumable format it should be provided as a REST API, serving the data as JSON.
  • Usually the open data we have is provided as CSV, which is reasonably structured (and is computable to a limited degree). Other open data might be in PDF or HTML formats, which is much harder to make computable.

##Problem 2

  • Users of data, such as app developers, or the Wigan Council Joint Intelligence Unit (JIU) would often want to look up and correlate data across multiple data sets, usually for a particular geographical area. This would mean looking up the postcode areas of interest against the first data set, then doing the same for all other data sets.
  • If you are trying to bring together a ‘dashboard’ of open data, this could mean looking up the same postcode reference against 10 or more data sets, possibly all in different formats, and manually collate the information.

##Idea

  • If we could encourage most Wigan Council open data to be published as a CSV (or convert/extract from existing formats into a CSV), then we could use a simple API wrapper to server the CSV data as JSON
  • There is already open source code that converts CSV to JSON ( https://github.com/project-open-data/csv-to-api ) although this converts the entire CSV file to JSON, so would require tweaking so that it responded with just the target data requested.
  • This ‘CSV to JSON API’ layer would make subsequent development of a JIU ‘dashboard’ for postcode level data much easier.

##Progress

  • This idea didn’t get picked up by a team on the day, and although I had intended to have a little go at it myself, I was busy with running the Hackathon and helping out Wigan Youth Cabinet with their online productivity apps (see other Project)
  • I have a good idea how I would set up a simple version or proof of concept of this, and perhaps we could work on it at a future Hackathon!

Marcus


#2

Hi Marcus

I’m really keen on the ethos of this idea as much as the idea itself. Providing your API wrapper idea works, JIU could be empowered by data from a number of currently untapped sources. Furthermore, publicly publishing certain sources of data via a web server for the tech community to directly utilise supports The Deal principles:
• Help communities to support each other
• Listen, be open, honest and friendly

Given better access to Council data, I’m sure all manner of exciting uses for it can be found.

Chris


#3

Hi Chris! Thanks for your support of the Hackathon and also for your support of the CSV2API idea.

Much of the information that the Council already has made available via HTML pages, could be made available via a parallel REST API, which would indeed transform the kinds of things that could be achieved at future hack days.

I’m happy to continue work on an implementation of CSV2API, but perhaps the best way to get this ‘infrastructural API component’ implemented, at scale, and to be trusted, is to ask the council to procure the additional functionality from Agilisys.

Happy to be involved in further discussions and to help where I can.

Marcus


#4

Hey - bit late to the party! Was looking for an API section on the council website did this progress? Can find a couple on brownfield land but not too many data sets …? https://theodi.org/ have done some decent work helping get data out…

cheers
John


#5

Hi John,

As far as we know, there is no API for any of the council’s main services. This is something we found to be a serious problem when we ran the Wigan Council Hackathon.

More problematically, the council team didn’t have any appreciation as to why they needed an API, what benefits it would bring, etc. They were slightly further ahead in terms of open data, but still not exactly a gold star.

The extent of Wigan Council’s ‘digital agenda’ seems to have remained in the fairly superficial realm of laughable one-day showcase events, endless twitter presence and cringeworthy #digileaders hashtags. I blame the self-appointed #digileaders themselves. I wouldn’t follow any of them into an SSH tunnel.

M


#6

I don’t think I would agree with the statement of the ‘digital agenda’ being superficial any more. Admittedly, when I first caught wind of it a couple years ago, I felt pessimistic by the councils approach and I think a lot of people did.

Though a few things have impressed me with how Wigan Council have handled that pessimism. They’ve actually started to listen to the comments and ideas people have in the community and they’ve been adapting to it.

The monthly Digital Taskforce meetings now contain a mix of council, community leaders (us included), business owners and even corporations. Now we have people with a varying mix of backgrounds in the same room, discussing, not just “how to get people online” but more of the bigger picture in how we can utilise the community organisations to educate and teach, important, marketable skills.

The Community Investment Fund is also another vehicle where the council are essentially saying “We know we don’t know what the community needs, so tell us”. It’s this type of thing that links in with teaching marketable skills and actually creating jobs within communities.

To the subject of the next Hackathon. The council have have approached us and asked, “what do you need from us?”. From this, we’ve got meetings planned to work out how we can make the data more accessible. They listened to us after the previous one, when we told them that their data isn’t actually open/accessible the way it should be.

There are people in the council that want to make a difference and are helping keep the feedback & adapt loop going. As long as that mind-set continues, I can only see bigger things develop out of it.