Search form

Become a Member Today Sign Up

METRICS & MECHANISMS TO MEASURE THE IMPACT OF OPEN DATA INITIATIVES

OPEN CITIES INDEX REPORT SERIES

 

METRICS & MECHANISMS TO MEASURE THE IMPACT OF OPEN DATA INITIATIVES

 
PSD’s Open Cities Index (OCI) is Canada’s first benchmarking study for municipal open data initiatives. The OCI launched in 2015 with 34 participating municipalities. In fall 2016, the second iteration of the OCI was published with 68 municipalities, representing 61 percent of the Canadian population. The 2016 OCI Report includes an overview of Canada’s Top Twenty Open Cities and national trends in municipal open data initiatives.
 
Leading up to the launch of the third edition of OCI in the fall of 2017, PSD will publish four quarterly reports exploring municipal open data challenges, solutions and best practices. Each themed report will include further data and analysis generated from the 2016 Open Cities Index study. After the publication of each OCI Report, PSD will convene a virtual round table of municipal open data practitioners to discuss the report findings, identify areas for additional research, and facilitate collaboration and the sharing of best practices across the country.
 
Contact us to learn how you can participate in an upcoming virtual round table discussion. 
 
Report 3) Metrics & Mechanisms to Measure the Impact of Open Data Initiatives – See below
Report 4) Models for Open Data Partnerships & Collaboration – September 20th 2017
 
 

METRICS & MECHANISMS to Measure the impact of open data initiatives 

 
In the first report in this series we explored the accessibility of datasets to Canadian municipalities of varying sizes and structures - it was evident that many local governments struggle to gain access to commonly requested datasets due to jurisdictional issues. In our second report, we demonstrated the significant capacity challenges facing local governments in Canada when it comes to building and sustaining an open data initiative. Despite these setbacks, municipalities have been able to work cross-departmentally to develop capacity and work with their respective communities to advance their open data programs. When it comes to measuring the impact of an open data initiative, however, most municipalities continue to hit a major roadblock.  
 
Through the 2016 Open Cities Index survey, PSD asked 68 Canadian municipalities about their ability to measure the performance of their open data programs. In an era of limited budgets and increasing public scrutiny, local governments must be able to demonstrate value in all programs and services delivered. Furthermore all departments must be able to track and report their progress toward achieving chosen objectives. For some areas of municipal service delivery, setting goals and measuring performance is relatively straight forward. Open data as a municipal service is fairly new, requires some technical knowledge to fully understand, and has a much less obvious impact on the taxpayer than say, transportation services. Consequently, local governments are struggling to develop meaningful, accurate and sustainable performance measurement frameworks for their open data initiatives. 
 

MECHANISMS OF MEASUREMENT

The results of the 2016 OCI survey clearly demonstrate the room for improvement in the area of "impact" for Canada's local governments. When asked "Has your municipality developed a mechanism to measure the impact of its open data program?" only 12 of 68 municipalities (18%) answered yes. Of the 12 cities indicating that they have a mechanism in place to measure impact, eight were in the top 20 percent and five were in the top ten percent. There are of course exceptions where some municipalities have excelled in implementing their open data programs without having a performance measurement tool or framework in place, and others that report having a measurement mechanism in place but have yet to achieve advanced maturity in open data. The former may have strong buy-in from staff and council, allowing for the sustained commitment to maturing an open data program without the ability to accurately measure impact (so far). The latter may represent communities that are starting with the impact problem, working to establish a robust performance measurement approach for open data before implementing the program.
 
For most of the 12 municipalities with some sort of measurement mechanism, simply tracking the number/type of data downloads serves as their only performance indicator (often using either an internal tracking system or Google Analytics). Council and senior staff will often want to know that the time and effort spent on gathering and publishing municipal data is resulting in public downloads, however, most communities lack a baseline or benchmark to know whether their download numbers are noteworthy. Furthermore, simply downloading municipal open data is only one piece of the puzzle - municipalities want to know how the data is being used. Instituting a survey that requires users to report how they intend to use the data before proceeding with the download may illicit some insights, however, creating new barriers to downloading the data does not sit well with most open data principles and there is no way of knowing that the user will actually use the data for his or her stated purpose. Municipalities may choose to post a voluntary impact survey to their open data portals, but the response rate may not be favourable. Leading communities in open data have taken a broader approach to the development of performance measurement frameworks, looking beyond just data downloads to both internal and external impacts of open data initiatives.  
 

CASE STUDIES IN MEASUREMENT

In Edmonton, ranking first in the Open Cities Index two years in a row, an Impact Map is used to track data usage. Going one step further, Edmonton tracks the effectiveness and efficiency of its open data program with a case by case value assessment of the initiative and its internal impacts. Finally, Edmonton surveys media hits related to its open data program, reporting on the percentage of favourable media coverage. The City of Toronto, ranking second overall in the 2016 Open Cities Index, has developed a set of performance measurement indicators through its Open Government Program. Developed by the City's Open Government Committee, these indicators measure progress toward achieving established goals under the four areas of Toronto's Open Government Program: Open Data, Open Engagement, Open Information and Open Culture.  Rounding out the OCI top three, Montreal has reached out to a local university to assist with the development of an open data impact assessment framework. The established framework will supplement Montreal's existing key performance indicators associated with its Smart City program. Working with a start-up accelerator, Montreal is also able to determine with some accuracy the number of local start-ups using the city's open data.   
 

MEASURED IMPACTS

Open data has the potential to improve government efficiency (eg. by reducing freedom of information requests - FOIs), strengthen transparency, contribute to economic development (eg. by supporting the growth of open data-based start-ups) and help solve community problems (eg. by providing researchers and community groups with access to free pertinent data.) So for those municipalities able to track the impact of their open data problems to some degree, what impact are they seeing? Three communities reported that open data has had a substantial impact on increasing government efficiency, while nine said that it had no impact at all. 34 municipalities reported that they had no way of measuring the impact of open data on government efficiency.
 
When it comes to strengthening transparency, only five municipalities reported that open data had a substantial impact, with 20 reporting a moderate impact and eight reporting no impact at all. Fewer communities (22) reporting not being able to measure the impact of open data on government transparency.
 
The most difficult impact of open data to measure appears to be the potential for economic development. Only one community has been able to track the positive impact of open data on its economy, with 45 municipalities reporting no way to measure. A further 23 communities report little to no impact on their local economies.
 
Finally, there is a small improvement in the ability of local governments to track the impact of open data on assisting with the identification of problems in the community. Four municipalities report a substantial impact, another four communities report a moderate impact, and 23 municipalities report little to no impact, with the remaining respondents having no way to measure.  
 
As seen with our previous reports, there is a clear correlation between the size of a municipality and the advancement of a community's performance measurement processes. All municipalities with a population over 500,000 have a way to measure the impact of open data on at least one of the above categories. There is also a correlation between a municipality's ability to measure impact (impact score) and its overall score in the Open Cities Index, as seen in the figure below.
 
 
Again, with some exceptions most communities will develop their open data programs uniformly across the three scoring categories of the OCI - Readiness, Implementation and Impact. As a municipality builds its open data team, processes and resources (readiness), the publication of greater quantities, quality and breadth of datasets will most likely accelerate (implementation). As the open data program matures, the municipality will have more information and insight needed to refine its approach to performance measurement (impact). Of course an open data budget doesn't hurt either. Figure 2 below demonstrates the positive correlation between a municipality's impact score in the OCI and the size of its open data budget.
 
 
Not all communities are in a position to secure an open data budget, but there are best practices being developed by municipalities of all sizes that can help kick-start performance measurement activities with little to no budget. We encourage you to share your ideas, challenges and solutions as we continue to facilitate the advancement of municipal open data initiatives across Canada. 
 

JOIN THE DISCUSSION

Please contact us to discuss how your municipality can participate in our virtual round table series and start advancing your open data initiative. Look forward to the launch of the 2017 Open Cities Index survey this summer, followed by the publication of the 2017 results in the fall.