Skip to main content

Fortnight notes: Data and Search team 21/09/16

Posted by: , Posted on: - Categories: Data and search

This is the Data & Search team's third set of fortnight notes. The idea is to keep an ongoing, public record of what we're doing. We'll iterate the notes with a view to making them better over time.

We'd appreciate other feedback to, or via the comments.

Here's what's been happening over the past 2 weeks:

Michael wrote a blog post about data modelling and beyond. Highly recommended.

On 13 September Dan presented to all Digital Service staff . Here are the slides.

Here is photographic proof that Dan presented to all Digital Service staff
Here is photographic proof that Dan presented to all Digital Service staff

As part of the presentation Dan started a pilot project to gather ideas, suggestions and problems related to data from across the Digital Service, to help collaboration. At the very least it's going to be a learning experience.

Emma approved the ODI report we commissioned and we expect to publish in 2 weeks with a blog post from Dan.

Dan and Samu did some interviews. 2 out of 3 were successful. One was an internal move - we're looking forward to Kunal joining the team as our second 'DataOps' engineer in a month or so.

Michael has been dissecting data from the Members' Names Data Platform and documenting it. Samu has working with Member data in tandem for consumption by the new website from our early iteration data service.

Michael had further domain modelling session with the Commons Public Bill Office on ping-pong and private members bills. Michael is publishing the domain models as they progress on GitHub.

Thanks to Aidan, Robert now has access to the Google search API and is writing a demo.

Robert is in Julie's recruitment as a product team. One of the things they have been developing is a Data Analyst / Analyst Programmer role. This role was one of the topics at our planning day on 14 September. It's proving difficult to define.

Also at the planning day we discussed what to work on for the new website with Jamie, Rebecca, Ganesh, and Kheira from the website team. Then in the afternoon we had a visit from Andy Dudfield of the ONS to talk about future collaboration. This was great, we're booked in to visit Newport in early November.

Aidan has made great progress on contracts, procurement, financial planning, and coordinating work with colleagues in Technology. The emerging consensus around how we collaborate with the Technology team is really encouraging.

Mat, Noel and David have completed 4 integrations for internal systems which are now in various stages of deployment. One fix saw an 85% reduction in job records marked 'Unknown Organisation'. David has also worked through 3 investigations for changes to systems as part of continuous improvement.

Thanks to Mike, we've introduced automation for restarting the Apache cache server in front of LDA, a component of This means uptime is expected to increase significantly. Actions are logged to a publically shared Google calendar.

Samu, Mike and Wojciech worked with Trung to estimate effort required to move to a different cloud host.

Wojciech & Mike showed proof-of-concept for GraphDB cluster spanning multiple cloud providers (AWS and Azure).

Samu's performance testing continues, having progressed from local machines to cloud (JMeter tests on VSTS).

The team did a show and tell on 20 September. Anya, Samu and Dan presented work to date for the Indexing and Data Management Section in the House of Commons. Audience feedback suggests that we're getting better with the clarity of our language.

Today (21 September) Michael is visiting Professor Michael Rush in Exeter to look at historical Member data.

It's been a good fortnight.

Sharing and comments

Share this page