2.11: Data in Context

HRC was not a job that sparked joy for me, though I did take the chance to explore the finer parts of Kentucky Bourbon. Author’s photo, 2024.

One of the easiest signs a leader is data literate is their ability to demonstrate critical thinking. Digitizing information is a plus, but amassing data isn’t going to really help you make better decisions if you don’t know how to approach a dataset. You need to know what the data says, and what it doesn’t; be able to know where it came from, and where you can get what might be missing.

Data always needs context. Nothing exists outside of some form of context, in particular, what is ‘normal’. One place that HRC missed on this was in how we reported promotion and command selection data. This is particularly problematic because just about every leader in SF has strong opinions about what will and will not get an officer promoted. The regiment is rife with old wife's tales on the subject, typically informed by data sets of one, but anecdote is not the singular of data. Field grade officers across the country are keen to follow the game trail of their boss, unaware of what really does, and does not, impact career progression.

This is not a knock against SF Branch, which has always been one of the best I’ve seen at keeping its officer population informed. Since my first one-on-one counseling back in late 2006, which was even before I’d actually passed selection, my SF branch assignment officers have endeavored to keep me informed and empowered about my army career, before I committed to having one. SF Branch pushes out regular newsletters and updates to ensure we all know when key events are coming up. The assignment officers travel around to do face-to-face counseling with almost every SF officer in the regiment. And they do this every year.

So I was very familiar with the promotion and command select slides that SF Branch used well before it was my turn to make them. I’d been looking at them since I was a captain. These slides dutifully reported a myriad of statistics about every year’s boards. Things like what percentage of officers were selected, and breaking down which look it was, and how many of their OERs were top or middle blocks.¹,²

One stat that caught my eye early on was the following: ‘90% of the SF officers selected to lieutenant colonel had a master’s degree’. While this was factually true, it was also missing context. When it became my turn to produce these slides, I took a deeper look at the available data. As I mentioned in my last post, HRC has troves of data going back decades, so there’s plenty of room to apply more data to a question.

In SF, on average, over 94% of the officers considered for promotion to lieutenant colonel possessed at least one master’s degree. What this meant was we should not be surprised that 90% of the newly promotable lieutenant colonels had advanced schooling, since an even larger portion of the eligible population had a master’s degree. If you reach into a bowl full almost entirely with green M&Ms, you shouldn’t be surprised to find yourself holding a handful of green ones.

Now I’m not arguing that getting a master’s degree didn’t benefit the officers. I have two of them. It was in grad school where I was given a thorough course on critical thinking and statistics, neither of which are taught as key subjects at Fort Leavenworth.³ What I am saying is without that context, people can easily draw false conclusions about what does and does not make a difference to a board.

As we received the results of the latest CSL board in the fall of 2018, I sat down and did some analysis on every argument I’d ever heard about what did and did not impact officers’ careers.⁴ Things like, which specific SF group an officer was from, where they went to ILE, or if they had a ranger tab. None of these moved the needle. Working for a general officer didn't either.⁵

Excerpt from ‘HRC SF Branch Newsletter, Vol. 12 (Mar 2019)’.

Perhaps the most pervasive old wife’s tale was ‘SF company command was critical to being selected for CSL’. A lot of leaders in SF fetishize company command as the key to making a successful leader.⁶ But for most of the army, a major’s key jobs are as the operations officer (S3) or the XO. On this, I actually agree with the army. Commanding an SF company was an incredible job, but it was my time as an XO and an S3 that taught me how staffs run, and how to motivate non-SF personality types.⁷ When we looked at the CSL results numbers for SF, just over a third of the 84 candidates had never been an S3 or and XO. Of these, only three of those were selected for CSL. Meanwhile, almost half of those officers who had never commanded an SF company were CSL selects, despite only being just over one-fifth of the total pool.

When the slating for commands came out, we did a deeper dive on the results, even taking the time to break down the word count and terms used in the senior rater narratives, which are by far the most impactful part of a candidate’s file. We transformed that data into an OER writing guide just in time for our higher commander’s conference that spring.

The commander’s conference hosts all of the ARSOF senior commanders, to include all five active SF group commanders.⁸ Given how impactful OER writing is to the future talent of the regiment, and the army writ large, the commander’s conference was the perfect venue to share our findings.

Or it would have been if we’d had more data literate commanders. The findings clashed hard with what almost all the commanders present believed in their heart to be true. A critical response to new data is not unexpected and should even be welcomed. But the response we got wasn’t a critique of our analysis, or our dataset. Instead, the more vocal of the assembled commanders questioned our impertinence.

‘How many OERs have you written?’. We tried explaining that while a colonel in the army had probably written almost 200 OERs, as assignment officers we’d reviewed over 6,000. But it didn’t matter about data, and it didn’t matter about expertise, because this was about the commanders.

Data illiterate commanders come up with an answer and then task their staffs to find the data that backs them up. But data fluent commanders ask good questions, often the questions no one else is asking. Then they go find the best data they can to answer that question, and they always stay open to more and new data to update their answers.

Only one of the five SF group commanders took the provided guide and adopted it that year.⁹ But, the OER guide outlasted that cohort of commanders, and saw gradual uptick in use over the years and has resulted in very high selection rates for SF officers both in promotion and in command.

That writing guide was my last project at HRC, and while HRC did not ‘spark joy’, I did take a lot away from my time there. I learned from the ORSAs there about how to better refine the questions I asked of data, and lots of tricks to help structure and move through Excel. I was indebted to the Toolkit team who never once rolled their eyes when I wandered down to their cubicles to ask for another new feature. And I learned the army has a dragon’s hoard of data at its fingertips, if it ever wants to know the answers to its questions. My next stop was back to Iraq, where I got to modernize processes by using some old (and in one case downright antiquated) techniques.

1. Officers are typically given three ‘looks’ at promotion. First, a BZ look (below the zone), then their PZ (primary zone) and then AZ (above the zone). Officers not selected by their third look can be released from the Army within seven months. There are some exceptions to this, but this is why the military’s promotion policy, set by congressional law, is called ‘up or out’.

2. ‘Top or middle blocks’ refers to the Senior Rater’s checkboxes. ‘Top block’ or MQs (Most Qualified) are the best rating, while ‘Middle blocks’ or HQs (Highly Qualified), are the most common. The two lower blocks, ‘Qualified’ and ‘Not Qualified’ are rarely used, but typically preclude promotion.

3. Fort Leavenworth is where the army trains the bulk of its field grade officers, in CGSC (Command and General Staff College).

4. Command Select List. The Army selects its Lieutenant Colonels, Colonels, and Sergeants Major for command via a centralized board process. CAP (Command Assessment Program) is part of this.

5. Unless that general officer was willing to apply undue influence on the board on your behalf. Task and Purpose has recently published an update on this incident.

6. Linguistically confusing, SF majors command companies. Our ‘company command’ equivalent to the regular army is as an ODA commander when we are captains.

7. SF personalities, with only the rarest exceptions, come pre-motivated. With 84 of the 87 billets in an SF company filled by SF soldiers, all you really have to do is keep them pointed in the right direction.

8. Army Special Operations, which includes SF as well as CA (Civil Affairs), PO (Psychological Operations, or PSYOP), the Rangers, and ARSOAC (Army Special Operations Aviation Command)

9. Another one of the five is a felon and in prison right now, so we’ve at least got a standard distribution of talent.