Search form


Become a Member Today Sign Up

Performance Measurement & Open Data

OPEN CITIES INDEX REPORT SERIES

 

performance measurement & open data

 

PSD’s Open Cities Index (OCI) is Canada’s first benchmarking study for municipal open data initiatives. The Open Cities Index Results: 2017 were released in November 2017, providing an overview of the state of municipal open data programs in Canada. The discussion did not end there. For the third year in a row, PSD committed to holding an OCI Virtual Round Table Series to bring together OCI members from across the country. The topic of the three Round Tables are (1) Open Data Governance, (2) Performance Measurement & Open Data, and (3) the Future of Municipal Open Data. The first Round Table was held on January 30th, 2018 and included open data experts from the City of Edmonton, the City of Toronto, and Carleton University. To supplement the Round Tables and encourage richer discussion of the topic at hand, PSD publishes a summary of the findings of the survey related to each topic one week prior to the Round Table. The first Round Table Report – Municipal Open Data Governance – can be read here.

The following summary will provide an overview of survey respondents’ answers related to performance measurement and open data. Performance measurement is important to any area of public service delivery in order to focus strategic priorities, allow for benchmarking, and demonstrate impact. In the context of open data, performance measurement is a significant challenge as definitions, standards, and mandates are still ill-defined for many public sector organizations. Furthermore, some municipalities have not yet achieved corporate-wide buy-in for an open data initiative, making it that much more difficult to spend time developing a meaningful performance measurement system.

This report is guided by three overarching questions related to performance measurement and open data:

  1. How do municipalities set, track, and report on meaningful key performance indicators of their open data program?
  2. What technology can be leveraged to collect data and interpret analytics related to your open data program?
  3. How can municipalities track reductions in Freedom of Information requests as a result of enhancements to open data programs?

 

Performance Tracking

PSD asked its 61 OCI survey respondents, “Does your municipality utilize a voluntary survey to ask of those that download municipal data, what they are using the data for?” The purpose of asking this question was to better understand if there is any mechanism (or mechanisms) that municipalities employ to identify how end users are making use of the data that they download.

 

85 percent of respondents answered “no” and 6 percent answered “yes” to offering a voluntary survey to those that download data. Another 9 percent answered “other”, which included respondents stating that they used other processes to learn of open data use, including asking via social media and gathering feedback at community outreach meetings. 

Without knowing how end users are using the data, how can municipalities tailor their open data programs to the needs of the end user? As cities continue to build out their open data program, knowing who is accessing the data will also help prioritize the order of data sets to be added to their portals and help develop targeted end user “personas” that provide context for the data in relation to the type of user (e.g. developer, educator, community leader, municipal staff etc.). Furthermore, decision-makers will often ask for this type of information to justify continued commitment and investment in an open data program.

On the other hand, the mantra of open data is that is must be open, free, and freely accessible. Introducing a survey – albeit a voluntary one – may be considered a roadblock for the unfettered access of open data. The participation rate also tends to be low and the results biased in voluntary surveys, so one might make the argument that the mechanism is not an ideal one to gather information about the end user. However, with limited alternatives, some info may be better than none. Further research around the success of social media campaigns and community outreach in gathering this type of information will shine a light on the viability of other alternatives.    

PSD also asked respondents if there was a process in place to track the number of open data downloads – another common metric asked of decision-makers. The most common system reported was Google Analytics (30%), often in conjunction with ArcGIS (15%). Other systems that were reported included WebTrends, Dynamic Downloads, Socrata, and Power BI. 39% of respondents answered “N/A”, indicating that they do not currently track downloads.

 

Key Performance Indicators

PSD asked, “Have you developed a mechanism to measure the impact of your open data program?” Of the 61 survey respondents, 52 (85%) reported “no”, while 9 (15%) reported they had a mechanism in place. Of those that responded “yes”, many had mature open data programs, meaning that they would already be reporting measures to their City Council and their Open Government Committee, in line with an established Open Data Master/Strategic Plan. Given this, one respondent who reported “no”, stated, “[Measurement] has been identified in our report and would be part of a future business case and project charter if City Council supports our request for resources to move our open data initiatives forward.”

One example of a reported mechanism of measurement was the International Open Data Charter. By adopting the Open Data Charter, municipalities can align and benchmark their open data initiatives against the Charter to measure the maturity of their open data program. Others cited KPIs as a mechanism of measurement. Finally, voluntary surveys were also reported as being a mechanism of measurement – understanding how, why, and to what extent end users are accessing the data can help build a case for impact. 

With a low percentage of survey respondents measuring the impact of their open data program, it is clear that more work needs to be done to develop maturity in performance measurement. Municipalities have made commendable strides in developing and solidifying their open data programs, but the next step is to begin measuring impact to strengthen the quality of their open data programs.

 

Measured Impact

PSD asked survey respondents about four types of measured impact in relation to their open data programs: Impact on government efficiency, municipal transparency, the local economy, and problem solving in the community.

 

1. Government Efficiency

In regards to government efficiency, PSD asked, “To what extent has open data had a noticeable impact on increasing government efficiency?” There was an overwhelming response reporting that staff efficiency was significantly enhanced. Specific examples of gained efficiencies included a reduced effort for data prep through easier access to data and reduction in data entry duplication. One respondent reported the advantage of having an open data program is that it ensures that “open” is indeed, open. “Information Technology Services have also incorporated open data review for all projects that may result in the creation of data products; ensuring that ‘open’ is considered during project development. This is saving staff time for data preparation as original products are already being created ready for publication to the open data catalogue.”

Furthermore, respondents also noted that an Open Data Portal allowed for cross departmental collaboration “that would have been impossible before”, limiting the “silo” effect that is common in government. Likewise, another impact reported was greater knowledge sharing. One respondent stated, “The greatest driver of our portal traffic is internal users. This indicates that the existence of our portal creates an optimized method for knowledge and information sharing across city divisions.”

 

Freedom of Information Requests

Another area of interest to understand the impact of open data on government efficiency is Freedom of Information Requests (FOI). Conceivably, enhancements to an open data portal (including more high quality datasets) could result in a reduction in the number of FOIs made by journalists, academics, and the public, as more data would be readily available for download, bypassing the need for an FOI. PSD asked survey respondents if they reported any reductions in FOI requests between 2015 and 2016, and if so, what was the percentage change? 19 (31%) respondents reported an increase, while 8 (13%) reported a decrease. With the exception of one outlier with a 100% increase since 2015, the majority of those that reported an increase reported an increase of under 36%. The majority of respondents that reported a decrease in FOI requests reported a decrease of under 20%.

Given the mixed results, and the variety of extraneous variables that would impact the number of FOI requests, further study is required to understand if there is a direct correlation between a more mature open data portal and a reduction in requests. For some municipalities, however, there are clear reports of advancements in open data resulting in reductions in FOI requests, and therefore an increase in government efficiency.

 

2. Transparency

One could argue that the fundamental goal behind open data is to enhance the transparency and accountability of government. But at the same time, the impact on transparency may be the hardest metric to quantify and measure. PSD asked respondents “To what extent has open data had a noticeable impact on increasing transparency in the municipality?” Examples provided of indicators of increased transparency included the following datasets being added to open data portals: municipal budget data, spending activity, council minutes and dispositions, council allowance, employee travel costs and purpose, and municipal election voting records. To summarize, one respondent noted, “Open data has driven the awareness of the value of transparency within the municipality as we are seeing a rise in the involvement of our departments in the open data program and offering data sets that provide insight into the operation of the municipality and its governing bodies.”

 

3. Local Economy

When asked whether their open data program has had a noticeable positive impact on the local economy, communities responded with varying answers. Some respondents noted monetary savings, including one municipality reporting a one-million-dollar savings from implementing a Transit API.

More broadly, the majority of respondents reported that the rise in the data industry has led to economic and business growth in their community. Specifically, one respondent noted, “Though we do not have a mechanism in place to measure the monetary and employment impacts our open data program has had on our local economy, through monitoring news reports and gathering word of mouth, the number of data centered employers is growing and furthermore opening up space for future growth.”

Likewise, respondents also stated that open data has informed and strengthened local businesses and community organizations. The majority of respondents provided examples of local entrepreneurs using municipal open data to build new businesses and services in the community. Some examples provided were Darkhorse Analytics and Truhome from Edmonton and Element AI from Montreal.

 

4. Problem Solving in the Community

Another impact question posed was, “To what extent has open data had a noticeable impact on identifying problems in your community?” Respondents were very affirmative that open data has brought greater attention to community issues. One respondent noted that the issue of homelessness in the municipality has become more explicit by being able to compare data from different organizations within the community. Another community impact of open data was that residents were able to know about traffic disruptions and thereby have the capacity to choose a different route. Another notable impact reported was that citizens are now less concerned about the transparency of their government. “People want [access to open data] because they feel there could be a problem with the way their tax dollars are managed.” Despite these legitimate and important impacts, this question received the most limited responses compared to questions related to government efficiency, transparency, and economic impact. 70 percent of respondents did not answer this question.

 

JOIN THE DISCUSSION

Clear here to participate in our virtual round table series and start advancing your open data initiative.

Performance Measurement & Open Data - April 24th 2018

PANELISTS
  • Debbie Verduga, Crime Analyst, Business Intelligence & Analytics Unit, Toronto Police Service
  • Jane Crofts, Founder of Data To The People & Data Literacy Evanglist, Melbourne, Australia
  • Invited: Wendy Carrara, Advisor, European Space Sector & Project Manager European Data Portal, Capgemini Consulting