AI Toolkit Assessment

Software teams often need to iterate on their processes and experiment with new workflows. For high stakes changes, it can be important to find a way to measure the impact of the change across a team, and across time.

One powerful tool a software team can use to do this is a pre- and post- measure. A pre- and post- design is a familiar and intuitive way to think about change. Simply put, a “pre” measure means that we attempt to start tracking something we care about before we try to change it. For example, perhaps you have a personal goal to improve your cardiovascular fitness. You decide to use the time it takes you to run one mile on a track as your pre- and post-measure, so before implementing a training program to increase your fitness, you time yourself running a mile. This is your pre-measure. After a month or two of focused training, you run a mile again and compare your new time – your post-measure – with your pre-measure time in order to evaluate your training program’s effectiveness at increasing your cardiovascular fitness.

Just like cardiovascular fitness, AI Skill Threat is complicated and multifaceted, and no single measure will capture its totality. But using a lightweight survey of AI Skill Threat both before and after interventions meant to mitigate it allows you to take advantage of the power of aggregating measures. By adding in the component of time, we give ourselves the power to create evidence that we can use to quantify the importance of what we’ve changed.

In the Generative-AI Adoption Toolkit, we’ve shared a Benchmarking Assessment comprising empirically validated, abbreviated survey items. This means that these items have been tested on a large number of people, and are designed to be stable and reliable. Given the practicality and lightweightness of this survey, you can use it to demonstrate the positive impact of activities like Learnathons and Pre-Mortems on mitigating your team’s or organization’s AI Skill Threat. 

Like all metrics with software teams, we recommend that tech leads, engineering managers, and other facilitators use pre- and post-measures thoughtfully. Our research on “thoughtful measurement” shows that developers benefit from measurement that: 

• Focuses on the process

• Celebrates effort

• Makes “invisible” learning & growth visible 

• Feels accurate and actionable 

• Respects complexity

Using mobile device? Take the assessment here.