One of my jobs in the data and search team is to work with data about Parliament's people. It’s mainly corporate data which comes from different integrated systems. This data about people is needed for internal purposes but doesn’t affect Parliamentary business.
We’ve blogged before about people data and what we do with it. In this post, I want to talk about what happened to this data during this year’s general election and what we learned from it.
We have lots of data
People data comes from multiple systems, some of these are:
- Members’ data from both Houses
- the pass office, which issues passes and security clearance for everyone that works in Parliament
- email addresses and network usernames
The data matching process is automated and it happens in the background. It has some rules around how it works:
- we match people data to email addresses and network usernames
- only the data systems from the pass office and the Houses can create, update or remove people
As you can see it’s very important for this process to work correctly. The data has to be accurate and the integrations must run in the right order. Our records are then shared with other systems that create, update or delete people’s records.
When general elections happen there are peculiarities that we have to prepare for. We need to be sure that the right people are in the system after the election.
During dissolution, a central team in Parliament have to mark all MPs as former MPs until confirmation of who’s been elected has been announced.
To avoid data loss, we turn off the feed with MPs' data. Otherwise, all MP records would have been automatically deleted and their records recreated again if they returned after the election, which would cause issues with other systems.
So we froze all MPs’ data in May this year, making our system ignore the fact that we were in dissolution. This way, we kept the system working as normal. Clear proof that ignorance is bliss and not just for humans.
9 June was a difficult day
When the results came in on 9 June, I decided to turn on the MPs’ data feed. It should have automatically removed both MPs who lost their seats and those who stood down. As well as adding all the new MPs. Unfortunately for us, it didn’t.
There was a problem with the feed. The server that hosts the API ran out of memory. That meant that the data we were getting was not complete so most MPs got deleted and the new ones didn’t get created.
Also, the data from the pass office came through with information and then some. It had created placeholder records with constituency names instead of the names of the MPs. So there was a Mr Saffron Walden, Mrs Oxford East, Miss Aldershot, to mention a few. Nice names for beauty pageant titles, but completely inaccurate for our MPs. We had expected to receive accurate data about people from the pass office, not unhelpful placeholders.
Even more complexity
To add more complexity, MPs’ staff can also become MPs. And this was the case for some of them in this election. In our system, some kept the same record but others had a new one created.
To recap, this was the situation that we found ourselves in:
- we ended up with an incomplete set of MPs
- we had extra records from the pass office with constituencies as people
- we had new MPs that originally were MPs’ staff and had duplicate records
- we had MPs’ staff with records even though their MP was non-returning
Brilliant! Data quality at its finest!
Unfortunately, there's no official source for MPs’ staff and we only receive information from the pass office. The Independent Parliamentary Standards Authority (IPSA) is only interested in MPs’ staff if they’re paid directly from the MP's budget, and not all staff are. That means that even if an MPs’ staff are removed from IPSA’s records, we still need to hold their data.
OK, enough already with all the negativity
At least we had an official list from the Commons Library which included IDs for returning MPs so it helped us identify which records to keep. That was something.
So how did we fix all of this?
- after the server was fixed, the Member feed took care of creating all new MPs
- some MPs had multiple records so we merged these manually and deleted the incorrect ones
- made sure all staff of MPs were moved to the right MP
- checked with the general election team when we had a green light to remove all the non-returning MPs from our systems
To get the system back to normal it took about a month, which is the time most non-returning MPs get as a 'grace period' for keeping their records.
I started working in this team three years ago, around nine months before the 2015 election. I can assure everyone that general elections are usually very stressful times within PDS. However, they clearly provide an opportunity to learn.
For the next election, we’ll need to gather more specific details from other teams about what they’re planning to do with their data, make sure we explain what impact it could have on our system, and try to negotiate a way forward if possible.
During elections, we need to plan a way to reduce the impact on the data as much as possible.
Read other blog posts from the data and search team.