DETAILS MATTER by Bob Ginsburg September 16, 2021
The Flaws in Data Driven Decision Making – All numbers are not equal in weight, meaning, or usefulness. Less staff limits ability to understand and act on “data” rather than just to plug in numbers.
Welcome to Details Matter a Newsletter about urban development, public finance, Transportation, and politics in Chicago and Illinois. You can subscribe/unsubscribe at (https://robertginsburg.substack.com/welcome). Get previous issues which mostly focus on dealing with the fiscal crisis at (https://robertginsburg.substack.com/archive). PLEASE FORWARD if interested.
The Flaws in Data Driven Decision Making – All numbers are not equal in weight, meaning, or usefulness. Less staff limits ability to understand and act on “data” rather than just to plug in numbers.
‘The devil is in the details’
A couple of weeks ago I talked about the decline in City employment and why this is a problem. The issue is not absolute levels of employment (though that is critically important in delivering essential services) but having enough employees with sufficient skills to devise and implement programs and services need in a 21st century economy. More employment, more training, more recruiting people to work in government as opposed to the private sector. Universities should also emphasize to students the value and importance of practitioners rather than just academics.
I was planning on discussing the City and County budgets as soon as the first proposals came out but I am putting that off as a release of a study from Pew (see graphic below) on the decline in government employment from pre-pandemic levels. Chicago and Cook County (and I imagine the State of Illinois) match the numbers especially when including unfilled budgeted positions. How to address this in an era of underfunded government operations is a challenge.
However, the real impetus for this newsletter came on September 10. The Illinois Department of Transportation announced ( idot.click/data-driven-decisions.) that it is seeking input as it develops a revised project-selection process intended to improve the efficiency and effectiveness of the state’s transportation system. Feedback from the public and stakeholders on the department’s enhanced data-driven approach to project prioritization can be provided on a new webpage as well as a webinar on Sept. 29.
They said that the new data-driven decisions tool will use information from IDOT’s nine districts, along with the various data tools the department currently uses, to prioritize projects that change capacity on the system. The revised project selection and evaluation system is supposed to increase transparency in the planning and programming process. I have listed their 5 goals and 13 criteria in a table at the end of the newsletter.
SO WHAT COULD BE WRONG WITH THE IDOT DECISION MAKING TOOL? DON’T WE ALL LIKE DATA?
On the surface, establishing goals and criteria for evaluating projects is a major step forward for IDOT. I recall one of my early meetings in 2011 as Administrative Director of the Cook County Department of Transportation when I called the senior staff together to discuss the proposed 5 year construction plan. When I asked what the criteria were for choosing which projects got funding and staff time to move them to funding in the following years, the only answer I got was that they “used the data available and best engineering judgement.” When asked about the policy goals of the County, I was told they were not part of the evaluation as there was not hard data associated with them. The process was changed and it has improved greatly at the County, but this “new” IDOT decision-making tool reflects a process very similar to what I encountered in 2011.
The IDOT proposal has fine goals and criteria as far as one can tell from the brief descriptions, but the proposal does NOT include how and when goals will be prioritized for specific projects and how it will affect funding decisions and allocation of staff time to work on projects. It is also centered on IDOT District staff which is dominated by engineers. Of course, some of my best friends are Engineers. They are critical for transportation projects, construction, building things in general. But, engineers, in general, depend on quantifiable parameters and tests where they understand the limitations - but they tend to resist prioritizing or even thinking about policy concerns and social goals which are not as quantifiable. I need only point out how long and difficult it has been and currently is for women to succeed in engineering companies. Thus looking at the proposed DOT decision-making tool, it is obviously weighted towards “quantification.”
For example, the first few criteria rely on experimentally derived data such as AADT (Average Annual Daily Traffic), or crash statistics, or commute times. Even identification of designated freight roads, intermodal yards, and location of major developments area hard data. On the other hand, there is a great need to rethink our transportation investments to address underserved communities which lack many transportation services. That is the stated goal of the Governor, Cook County Board President and the Mayor of Chicago. We have transit and transportation deserts in many areas – both rural and urban – so how readily will the engineers (as designated in the tool description) evaluate the hard numbers as compared to areas with a lack of information.
These “soft” factors (unfortunately using the engineering terms for data) show up in the environmental impacts/livability and regional concerns goals. How are these data going to be compared? The comparison is not simply done by plugging numbers into an equation or comparing numbers. How will they compare and prioritize improvements on Lake-Cook Road with an ADT of 40,000 vehicles per day versus a Harvey, IL (which unfortunately ranks high on socio economic factors) but does not have high traffic levels, measured pollution, etc. It does have some truck traffic but with many other needs (along with its neighboring communities). Those needs include more access to public transit, etc.
There are alternatives that provide transparency and a clear process that can include soft data justifications. I include below a proposal requested by IDOT in 2006 for an agency wide decision-making process that would begin to change the culture of how projects are conceived and justified. The core of the proposal (derived from a review of several other state programs) was a set of 8 program and process goals that could be applied at each stage of the development process. How each criterion was addressed and prioritized at each step would be clear – even if it was not considered, it would still have to be acknowledged and a reason given. In time all staff (engineers, policy, management, and even political/legislative) would have to go through and acknowledge the goals for each project. This is one of many options but the currently proposed tool does not appear to change the underlying culture and decision-making process at IDOT.
“Data driven” policy is not limited to transportation and its uses in other areas suffer from the same problems. For example, the new head of the Chicago Public Schools touts his “Data Driven” approach but sometimes the straight data needs to be overridden to address long-term problems.Just like investing in the underinvested south suburbs of Cook County may not lead to quick reductions in ADT, congestion, or even emissions, in the long term they will produce those benefits and others that are perhaps not as easily quantified. The same with schools and investments that will lead to more enrollment and better outcomes in future years. Public investments should look to longer term benefits not very short changes. Any process has to balance the hard data with the policy and equity issues.
WHERE IS THE PRIORITIZATION OF EQUITY?
Finally if we are to make equity a priority, we have tostart doing things differently? If IDOT is serious in addressing how projects are identified, prioritized and implements AND in responding to the legislatures concern on how the Capital Bill funding will be directed, then this tool needs to be refined and include an explanation of how it will be used. A recent presentation by the Metropolitan Planning Council at a Transportation Research Board conference on equity proposed some criteria for including equity in planning and is listed below. It would be a good start for IDOT.
MPC Transportation Equity Criteria. (NB. Not all projects address all goals but all goals should be considered.) Equity evaluation considers if a project is:
•Concerned with the distribution of benefits and burdens of transportation projects, plans and policies between individuals and groups that differ by race, income and ability;
•Designed to protect and increase the benefits—with an emphasis on accessibility—for historically marginalized populations, especially low-income communities of color;
•Allocating resources based on communities’ needs, with the aim of correcting existing differences and removing the effects of past discrimination;
•Providing effective opportunities for disadvantaged populations to participate in the transportation decisions that will affect them.
IDOT five goal areas and thirteen selection criteria. (September 10, 2021)
Goal: Traffic Operations/ Congestion
Criteria: Annual Average Daily Traffic (AADT)
Criteria: Change in Annual Vehicle Miles Traveled (AVMT)
Criteria: Travel Time Index (e.g peak travel time/free-flow)
Goal: Safety
Criteria: Crash Frequency
Goal: Economic Development
Criteria: National Highway Freight Network
Criteria: Major Development
Criteria: Intermodal Accessibility
Goal: Environmental Impacts/ Livability
Criteria: Environmental Justice : located in an area identified as minority population, low income, or both based on the measures identified by the Illinois Environmental Protection Agency
Criteria: Level of Environmental Impact Analysis Required
Criteria: Equity
Criteria: Resiliency
Criteria: Emissions
Goal: Regional Rating
Criteria: Subjective portion allowing local and regional input, to consider factors which may not be shown in the data – ALL evaluated and obtained by IDOT staff
Alternative Project Evaluation model requested by IDOT in 10/2006 but not adopted
Framework Goals and Planning Stages
• Eight (8) Planning and Prioritization Goals: the language of each Goal attempts to better convey each Goal’s intent as well as provide department talking points for IDOT. For instance, the Goal language “Support Industry-Specific Economic Development Strategies” has a clearer intent than simply “Economic Development”
• Performance (6): consider the Technical and Policy Merits
• Process (2): In parallel to Performance evaluate real ability of funding and financing and geographic equity and local priorities
• Different Planning Stages: Policy/Program Goals (Statewide), Investment Priorities (e.g. Regional or Broad Local Program decisions), and Project Selection (e.g Local TIP project selection)
• All Criteria apply at each level or Stage of planning. HOWEVER,
• Relative Weighting of Goals will Vary at Investment Priorities and Project Selection Stages
* Questions and Performance measures vary with projects and at different Stages
Framework Goals
Performance Goals
Support Industry-Specific Economic Development Strategies
Enhance Social, Demographic & Geographic Opportunity
Coordinate and Integrate with Land Use & Existing Transportation Networks
Improve Resilience & Congestion
Preserve & Improve Existing Assets
Reduce Environmental Impact
Process Goals