This is the Data & Search team's sixth set of fortnight notes. The idea is to keep an ongoing, public record of what we're doing. We'll iterate the notes with a view to making them better over time.
We'd appreciate any feedback to @dasbarrett, barrettd@parliament.uk, or via the comments.
Here's what's been happening over the past 2 weeks:
On 20 October we had an event at Newspeak House about Parliamentary data. Michael set this up and Dan spoke. Henry put together a summary of what happened. We were really, really happy with how it went. We got some constructive feedback, and several ideas for what to do next. We also have an open document for submitting ideas and requests for data as a result of the event. Many thanks to Ed (the founder of Newspeak House) for his support and advice.
On 24 October Kunal joined the team as our second WebOps (data) engineer.
Dan did a conference call with colleagues from the Scottish Parliament, talking about our approach to the new data service and the challenges we face. Many of these challenges are shared.
Robert began work on sorting through user needs research by Kheira for search. He also wrote a set of slides describing the team's current position and understanding of our (fairly complex) search landscape.
Robert continued to work on PHP wrappers for Google API that would be suitable for Microsoft Azure Functions, and deployed via Dropbox.
Michael, Anya and Silver had an excellent session with the House of Lords Bill Office modelling the passage of private and hybrid bills.
Julie worked on getting two new roles for the team advertised in time for the Silicon Milkroundabout job fair. The Digital Service will have a stand on 27 November. The roles are 'Data Analyst' and 'Test Automation Engineer'.
We had a good planning day on 26 November. Chris, our new Junior Developer who starts on 7 November, joined us. We covered most of our agenda, including a first pass at documenting our working- and decision-making practices.
From Wojciech and Samu's work, data in our new data service now accommodates "find your MP". This means we have cloud-based orchestration (scheduled data retrieval and transformation in Azure Functions and Logic Apps) from two sources covering the following Parliamentary business entities: People (Members), Houses, Parties, Constituencies, Seats, Sittings.
Robert, Samu and Anya advised a project team looking at new Library management systems about integration with search.
Dan and Samu met colleagues from Technology to review the design for the Data and Search live hosting environment. There was general agreement, and the offer of helping the team with Azure expertise.
Samu met with Rupert (Enterprise Architect) to provide input to the Digital Service Technology strategy.
Samu restarted collaboration with Rebecca and Giuseppe from the website team. We will prioritise work on our API to take as much burden off them as possible. In the future they'll probably create the first SDK for the Parliamentary data platform. Samu says this will be awesome.
Dan met with new colleague Steve to talk about how we can kick of ongoing and sustainable user research for our data services.
Samu, Robert, Michael and Ganesh went to the ODI summit on 1 November. Blog post to follow.
David successfully implemented new fields in our internal 'people directory' system: Preferred First Name and Preferred Last Name.
Data fixes were applied to mitigate incorrect or incomplete Job information on 'people directory'. This reduced Unknown JobTitles by 50%, Unknown Org by 70% and Unknown JobTitles + Unknown Org by 100%.
Noel and David began work on obtaining updates from Active Directory (also now including the preferred name fields).
Mat implemented a 'Blacklist' to prevent non-person records from different systems being created on 'people directory'.
Mat agreed a modification on the data integration for ACT (internal training system) so that all the people that work in Parliament appear on ACT whether they have or not a network account. This makes reporting easier.
Aidan met with Emma (our Director) to discuss collaborative work with academic institutions, both in terms of developing new data services and also outreach and engagement across Parliament to spread the word of linked data and the semantic web.
Aidan worked with Samu and Michael to create statements of work for the University of Southampton covering ontology development and a technical review of our new data service.
Aidan also continued to work on procurement.
Our colleague Paul in Business Systems put lda.data.parliament.uk behind our CDN on the weekend of 30 November. There has been 100% uptime so far. However, we experienced a new, separate issue with data.parliament.uk performance. Our CKAN instance (the data catalogue) went down over the same weekend. Mike has been investigating the root cause and looking to automate maintenance. As part of the investigation, Samu and Mike identified some orphaned cloud resources that could be turned off to reduce cost, and the need to increase the computing resources for CKAN.
Samu, Mike and Kunal worked on an ongoing issue with our internal 'Parliamentary Search'.
Dan continued to work with colleagues from Emma's directorate and beyond on aligning the work of the website, data and search, and business systems teams.
Dan did coaching and advice-giving to a variety of people.
Samu, Robert, Mike, Wojciech and Dan have all taken holiday in the last two weeks.
Robert made a slackbot to remind us to update our fortnight notes channel. It's very diligent.