Search form


Become a Member Today Sign Up

2017 Open Cities Index Methodology

Similar to the 2016 Open Cities Index methodology, the 2017 methodology adapts the model of the Open Data Barometer for application at the local level in Canada. The Open Data Barometer study, conducted by the World Wide Web Foundation, successfully benchmarked national open data initiatives around the world in 2013 through to 2016.

In establishing the Open Cities Index, PSD first reviewed all relevant open data literature including studies from across Europe and North America. In addition to the Open Data Barometer, these studies included POPSIS (Pricing of Public Sector Information Study) conducted by the European Commission, McKinsey Global Institute’s Open Data Study, and OKFN’s Open Data Census.

 

 

THE SURVEY

Seeking to gather highly accurate and valid data on municipal open data initiatives, PSD opted not to pursue the crowdsourcing model for data collection used by the Open Data Census. PSD, in partnership with Canada's Open Data Exchange (ODX), set out to create a survey that municipalities would complete themselves and return to PSD for validation. The Open Cities Index survey was created using Survey Monkey. The 2016 Open Cities Index survey included more complex questions that allowed for a higher level of detailed data to be collected from municipalities. In 2017, additional questions have been added to the survey to keep pace with the evolution of open data maturity across Canada’s local governments.   

The survey is separated into three sections based on the key elements of an open data initiative: Readiness, Implementation, and Impact.

 

READINESS

The readiness portion of the survey is designed to evaluate the extent to which a municipality is ready to build its open data capacity and foster positive outcomes through their open data initiative. The entire readiness section is marked out of 64 possible points.  This score out of 64 is then converted to an overall score of 25, and is worth 16% of the overall grade.

For each scored question in this section, 2 additional points are alloted for each incremental movement towards maturity. For example, 0 points are alloted for a community that indicates that they have no open data committee in place, all the way up to a maximum of 8 points alloted for a community that indicates that they have a committee in place that meets monthly. The full list of scored questions for this section is outlined below. 

              

 

IMPLEMENTATION

The Implementation section of the survey was designed to measure the extent to which a municipality has fulfilled its open data goals and ultimately, what data it has posted online. This section looks at 36 specific datasets, up from 32 datasets reviewed in the 2016 Open Cities Index.
 
  • Government budget
  • Election data
  • Council expenses
  • Council voting records
  • Census data
  • Lobbyist information
  • Public facilities and structures
  • Company register
  • Municipal permits
  • Code enforcement violations
  • Construction contracts
  • Procurement contracts
  • Property assessments
  • Park/Beaches inventory
  • Public transit
  • Real-time transit
  • Traffic volumes
  • Traffic accidents
  • Road closures
  • Service requests (311)
  • Zoning (GIS)
  • Base GIS data (roads, addresses, etc.)
  • Restaurant inspections
  • Health performance
  • Education performance
  • Crime statistics
  • City services
  • Bylaw infractions
  • Environmental services
  • Air quality
  • Recreational programs
  • Web analytics
  • HR data (NEW)
  • Safety data (NEW)
  • Street tree data (NEW)
  • Culture data (NEW)
     
All 36 possible datasets are assessed against the following 11 variables in order to ascertain the quality of the data:
 
 
Points are allocated for all published datasets (to a maximum of 10 points per dataset reflecting the number of the above variables satisfied by the data).  Zero points are allocated if a municipality does not have access to that particular dataset. The variable “The Data Exists” was added to the 2016 Open Cities Index in order to capture the types of datasets that are commonly unavailable to municipalities across Canada. In total, a municipality has the opportunity to score 360 points in this section. This score is then adjusted and converted to an overall score of 100, and is worth 68% of the overall grade.

 

IMPACT

For each multiple choice question in this section (five potential answers), 2 additional points are alloted for each incremental movement towards maturity. For example, 0 points are alloted for a community that indicates that they are not collaborating with their local school boards on open data initiatives, all the way up to a maximum of 8 points alloted for a community that indicates that they have substantial collaboration with their local school boards.  In this section, there are also two Yes/No questions. For an answer of No, 0 points are given, while 1 point is given for an answer of Yes. The full list of scored questions for this section is outlined below. 

 

Overall, the three sections are weighted such that the total score is 150. The weightings are as follows:

VALIDATION OF SURVEY RESPONSES

For 2017, the PSD analytics team will audit all of the survey responses within the each section for accuracy. In order to facilitate the validation process, survey respondents are asked to provide the links to all datasets (and links to reported plans, policies etc.). PSD will also incorporate these links into the national open data benchmarking tool, allowing OCI Premium Members to browse municipal datasets throughout the year.

 

UPDATES IN 2017

In order to provide municipalities with even greater insight into the relative progress of their open data initiatives, several changes were made between the 2016 and 2017 iterations of the Open Cities Index. In the readiness section of the survey, scored questions were added pertaining to the governance of open data initiatives and the extent to which municipalities are providing educational resources related to open data. In the implementation section, four new datasets were added to the list of eligible scored datasets: HR data, Safety data, Street tree data, and Culture data. Survey respondents are also asked to provide links for all reported datasets. Finally, in the impact section of the survey, additional unscored questions were added pertaining to the number of downloads/views of datasets, the reported change in freedom of information requests from 2016 to 2017, and the emergence of civic-tech groups and the development of crowdsourced citizen platforms. New scored questions added to this section include an assessment of the level of collaboration with regional innovation/business centres and with post-secondary academic institutions, and an assessment of case studies/personas created to provide context for municipal datasets.

 

THE RESULTS

The deadline to complete the 2017 OCI survey is September 8th 2017. The survey validation process and analysis of data will take place over the fall, with the 2017 OCI results - including the new OCI Top 20 - to be announced October 2nd 2017.

Following the launch of the 2017 results, PSD will host another round of quarterly virtual roundtable sessions with municipalities of various sizes from across the country, providing the opportunity to discuss open data challenges with municipal practitioners in the field. OCI Members are able to attend our virtual roundtable sessions throughout the year. OCI Premium Members are able to access our national open data benchmarking tool, allowing for the most up to date comparisons of municipal open data initiatives in real time.

Based on the 2017 OCI results, PSD will again award Canada’s Top 3 Most Open Cities, as well as announce the winner of the Most Improved Community Award and the Small Community, Big Impact Award. New this year, PSD will be launching nominations for the Municipal Open Data Champion Awards, recognizing excellence in leadership, strategy and collaboration across Canada. The nomination process will be announced shortly.   

Please contact us if you have any questions about the OCI Methodology or to inquire about your community’s participation in the 2017 survey intake.

info [at] publicsectordigest [dot] com

519.690.2565