Chapter 9: Assess your outcomes
Abstract
The OILRIG checklist is the focus of this chapter. The items enable a police commander to consider if his or her objectives were met, if the project was implemented correctly, and if any lessons learned resulted in a successful project. If not, there are ways to identify if intelligence gaps will enable a better operation next time, and if the goals from earlier in the PANDA process should be revised. An example case shows the importance of understanding seasonality and having a counterfactual in understanding operational effects. Outcome and implementation (process) evaluations are explained. A section on learning from less-than-perfect outcomes explains how to build on a program. Finally, an FBI-led gang takedown and a tool that can help assess the value of these types of operation are described, with details of where to find the spreadsheet. A bullet point summary completes the chapter.
Headings and sub-headings
The OILRIG checklist
Learning from failure
The ABC spreadsheet
Chapter summary
Additional links
The Campbell Collaboration has a range of downloadable resources around Scared Straight, concluding simply that "Scared Straight interventions cause more harm than doing nothing".
While a little quantitative and a bit nerdy, there is a blog post on jratcliffe.net that explains why we should not compare year-to-year crime counts. Most of the writing on seasonality is behind paywalls in academic literature, but there are one or two open-source summaries.
Displacement is a frequent concern of focused police operations. While there have been more recent studies, a brief summary of a 1994 literature review is available from the New South Wales government. Rather than expecting displacement, we should as frequently expect a diffusion of benefits.
Vignette author Jonas Baughman is an NIJ LEADS scholar.
Matthew Syed's book Black Box Thinking is available on Amazon.
A little more about how James Dyson got to where he is, is available from Inc.
The FBI have specific details of the various indictments around Operation Thumbs Down both here and here.
The ABS spreadsheet is available from this webpage.
Figures
Click here to download a full version of this picture in PowerPoint format.
Want to use this image in your work? Click here to understand how and when you can use images on this website.
Vignette authors
Jonas Baughman
Sergeant Jonas Baughman is a 14-year veteran of the Kansas City, Missouri Police Department (KCPD). A native of the Kansas City area, Sergeant Baughman joined the KCPD after obtaining a B.A. in psychology from Creighton University. He has held assignments in patrol, investigations, and crime and intelligence analysis during his tenure. Sergeant Baughman quickly found crime analysis to be his professional passion, and more than half of his career has been in positions related to crime or intelligence analysis. Past assignments include having served as a crime analyst, creating and supervising the KCPD's first Real-time Crime Center team, and directing a squad of detectives tasked with gang intelligence. He is currently assigned to the Office of Chief of Police where he is responsible for strategic analysis and evaluation of crime-reduction strategies and similar projects, among other duties.
Andy Parkes
Chief Inspector Andy Parkes joined Leicestershire Police 2001 after an 8 year career in the pharmaceutical industry where evidence based practice was the norm. He has enjoyed a variety of posts as sergeant; 24/7, proactive team, neighbourhoods, intelligence, and community safety, before being promoted to Inspector. It was as a local policing unit commander in 2012 that he first got to practice evidence based policing with his alley gates project, using this as the basis for a Masters dissertation. In mid-2015, after a dalliance with the detective world and cybercrime, he got ‘back in black’ and into his current role of Priority and Resource Commander dealing with firearms, public order, CBRN and critical incident command alongside 24/7 policing.
He still endeavours to fully embed an evidence-based approach to problem solving within policing, delivering inputs locally and nationally and retains passionate about this as well as delivering enabling mobile technology to front line officers.
Chapter-related review questions
Chapter 9: In the OILRIG checklist, the O represents what?
A. Outcomes achieved?
B. Objectives achieved?
C. Opportunities exploited?
D. Outputs achieved?
Chapter 9: In the OILRIG checklist, the first I represents what?
A. Information gained?
B. Implemented as planned?
C. Issues understood?
D. Intervention conducted?
Chapter 9: In the OILRIG checklist, the L represents what?
A. Lessons learned?
B. Limits acceptable?
C. Limitations accepted?
D. Length of project acceptable?
Chapter 9: In the OILRIG checklist, the R represents what?
A. Response acceptable?
B. Reactive policing effective?
C. Responders acceptable?
D. Results acceptable?
Chapter 9: In the OILRIG checklist, the second I represents what?
A. Issues understood?
B. Information gained?
C. Intelligence gained?
D. Intervention conducted?
Chapter 9: In the OILRIG checklist, the G represents what?
A. GOALS to be revised?
B. Ground commander acceptable?
C. Ground commander effective?
D. GOALS to be repeated?
Chapter 9: Another term for an outcome evaluation is what?
A. Objective evaluation
B. Implementation evaluation
C. Impact evaluation
D. Planning evaluation
Chapter 9: Crime counts can be constantly shifting over time due to three factors. Which is NOT one of the three mentioned?
A. Trend
B. Predictability
C. Seasonality
D. Variance
Chapter 9: What is another name for an area that serves as a counterfactual?
A. A control area
B. A broken windows area
C. A buffer area
D. A target area
Chapter 9: What is another term for an implementation evaluation?
A. Process evaluation
B. Problem evaluation
C. Outcome evaluation
D. Objective evaluation
Chapter 9: What is NOT one of the suggested strategy for conducting an operational post-mortem?
A. Set time limits and be realistic about what you can achieve in the time allocated.
B. Do not dismiss any feedback out of hand, even if it is unfair or bias.
C. Choose a suitable venue that limits distractions and in which people can be comfortable.
D. If you think your rank might intimidate participants, use it to demand more input.