on stratfor

How Grading Our Work Makes Us Better Forecasters

Amelia Harnagel
Director of Research and Collections, Stratfor
8 MINS READNov 19, 2018 | 17:10 GMT
When we are smart and critical in our analysis, when we are diligent in applying our geopolitical lens to the world, we can begin to bring the future into a clearer focus.

As you might expect, grading our forecast is an incredibly time-consuming process. But it is perhaps the most necessary thing we do as a company.

Editor's Note

We originally ran this On Stratfor column by Amelia Harnagel on Dec. 15, 2017, ahead of our 2017 forecast report card. As we put the finishing touches on our 2019 Annual Forecast and prepare to publish our 2018 forecast report card, we are reposting this column. It remains as pertinent now as it did then, and serves as a reminder to ourselves and our readers of the rigor we apply to the forecasting process and geopolitical intelligence in general. Some dates have been changed to reflect when the latest report card publishes. 

In action movies, "don't look back" may be good advice, but it doesn't fly here at Stratfor. As a geopolitical strategic forecasting company, we naturally spend a lot of our time looking forward, using our analytical framework to tell readers and clients what really matters in the long run. After all, our company was founded on the idea that with an advanced understanding of geopolitics and the employment of empathetic analysis (by which we envision what it may feel like to be in a world leader's shoes), we can anticipate the path that major global trends will follow. But for us to write intelligently about the future, it's just as important that we look at the past — especially our own past performance.

What It Means to Forecast

Over the years we have developed a lexicon around the process we call strategic forecasting. If you walk through the Stratfor offices or sit in on meetings among analysts, you'll hear us toss around terms like "directionality," "inflection point," "constraints," "leverage" — and of course, the all-important term "geopolitics."

Geopolitics is one of those buzzwords that gets thrown around constantly in various contexts, always with a different intended meaning. At Stratfor, we consider geopolitics to be both a discipline and a tool in our strategic forecasting work. As a discipline, geopolitics is the study of the "intersection between place and people, or more specifically between place and the nation, and the impact they have on one another." As a tool, it's a lens we can employ to prioritize and distill the trends that will most affect a given space.

Indeed, when you endeavor to look at the world through the geopolitical lens, things change. What a world leader says becomes a lot less important than what their government actually does, for instance. And when we are smart and critical in our analysis, when we are diligent in applying our geopolitical lens to the world, we can begin to bring the future into a clearer focus. That's what our decade, annual and quarterly forecasts are for. In them, we lay out the constraints that major powers will face, the critical dynamics that will play out among countries, and the general shape that the future will take on a geopolitical level.

Of course, it would be impossible for our forecasts to cover everything. If they were so exhaustive that they addressed every domestic issue in every country, our forecasts would be at odds with our purpose and they wouldn't serve our readers. After all, our job is to clear away what doesn't matter and tell you what really does — and why. We live to put your world in context.

The Constancy of Change

But there's a reason we prefer to call our work "forecasting" and not "predicting"; over the course of a year, trends in geopolitics change and evolve. Personal choices result in international repercussions. Black swan events play out. That's simply the nature of the fascinating, complex world we live in. And so while we begin each year with an annual forecast, we use our quarterly forecasts to advance and adjust that narrative throughout the year. (Similarly, we use the annual and quarterly forecasts to do the same with our decade forecast.) And on an even more granular level, every piece we write — from a short snapshot to a long assessment — serves the function of updating our forecast and assessing whether an event challenges or confirms trends we've previously identified.

The deeply interconnected nature of our content allows us to constantly adjust our perspective and refine our thinking, but it also opens us up to compounding errors. Because the forecasts are crucial to our intelligence methodology, their accuracy is paramount. After all, no one wants a structure built on a faulty foundation.

Keeping Ourselves Honest

Early in the history of the company, Stratfor established a rigorous self-assessment procedure to guarantee course correction on a regular basis. Behind the scenes, we call this our report card, and it's our acknowledgement that the best way for us to improve our ability to look forward is to turn around and take a look back.

Here's how it works: Each December, Stratfor's team of researchers from around the world divides the yearly forecast based on geographic expertise and grades it. (As a way of minimizing bias and guaranteeing intellectual honesty — and with the always-beneficial side effect of encouraging interdepartmental cooperation — we don't give this task to our team of analysts.) The researchers then dive into the document and grade every assertion that it makes, line by line. We don't grade what we meant. We don't grade what was said in a previous draft or in a planning meeting. All that matters are the words that are published in the final forecast.

As an example, let's say our yearly forecast includes the following sentence: "Beijing can be expected to increase cooperation with Moscow this year on energy projects, military coordination (particularly in the Arctic) and cyber-technology." Come December, our research team will go back through a year's worth of information to evaluate each of the three assertions made in that one sentence, determining whether it's a hit, a miss or a "wait and see" item. Ultimately, then, our researchers give each section of the forecast a letter grade from A+ to F.

The Hits, the Misses and the Reason We Care

As you might expect, grading our forecast is an incredibly time-consuming process. But it is perhaps the most necessary thing we do as a company — and not just so we can give ourselves due credit for the things we got right. We're pleased to say that we end each year with more hits than misses. For example, this year, as the threat from the Islamic State has abated in Iraq, we've seen increasing conflicts between Kurdish militia forces and Arab Shiite forces, particularly around Kirkuk, just as we forecasted.

But because we have such high standards, and because we are endlessly inquisitive and rigorous in our work, it's the items we were wrong about that we spend the most time examining. We sit down as a team and we walk back through the steps that led us astray. We deconstruct our logic. We erase any assumptions we may have had. And we rebuild our framework so that it's better than it was before. In this way, we can turn every "miss" into a lesson learned, and we can not only avoid repeating the same mistake, but we can also strengthen our future forecasts and develop more advanced analytical techniques.

Sometimes a "miss" means we say something will happen and then it doesn't happen. Take, for example, our 2016 forecast on the Brexit vote, in which we anticipated that the United Kingdom would land on the side of "Remain." As we noted in our 2016 report card, this incorrect forecast prompted intense introspection and allowed us to adjust and improve the way we approached inertia as a factor in the playing out of overarching trends. Then we marched forward into 2017 armed with the wisdom we gained, and we delivered more nuanced, informed forecasts for other European elections in France and Germany.

More often, our misses come in the form of blind spots — that is, events that we simply did not have on our radar. The swell of protests leading to a pro-Western government and subsequent Russian military action in Ukraine during 2014 is a good example of this. The consequences of these events required us to step back and reevaluate Russia's relationship with the West, and it would be difficult to overstate how much we learned from them. If we hadn't held our feet to the fire and evaluated where we missed the mark, we wouldn't have developed the current, improved framework that we use to examine Russia-West relations, and we wouldn't have been able to offer nearly as much insight into Russia's motivations over the past couple of years.

Confronting our mistakes can be uncomfortable, especially since we always strive to balance respect for our colleagues with a shared dedication to accountability. But failing to be self-critical as a team will inevitably lead us to wrong conclusions, undermining our value proposition to our subscribers and clients. And frankly, by being realistic about what we've gotten wrong, we're far more likely to get things right in the future. With every mistake comes wisdom gained.

On Wednesday, Nov. 28, we'll be sharing our 2018 forecast report card with you, our readers. In everything we do, we strive to be open and honest — and that extends to the transparent acknowledgment of both our success and our failures. With the ever-churning news cycle and never-ending sources of information, we know how much competition we face for your time and attention. And in the era of real and perceived fake news, it is more important than ever that we critically examine the work we've done in the past as we set the course for the coming year.

Article Search

Copyright © Stratfor Enterprises, LLC. All rights reserved.

Stratfor Worldview


To empower members to confidently understand and navigate a continuously changing and complex global environment.