Forecasting global trends and developments can prove challenging even when you have the best information available. Stratfor leverages the study of applied geopolitics, tools of intelligence analysis and public report cards on our work to constantly improve our forecasting process. The team at Good Judgment Inc. seeks to directly quantify their forecasts with numerical probability assessments.
In this episode of the Stratfor Podcast, we’re joined by Good Judgment’s Philip Tetlock and Warren Hatch as well as Stratfor’s Rodger Baker and Mark Fleming-Williams to explore how qualitative insight and quantitative rigor can be applied in unison to improve forecasting of global developments. This is part of a larger collaboration between Stratfor and Good Judgment as we explore the future of European integration.
Related Reading
The Numeric Problem by Rodger Baker
The Numeric Solution by Dan Gardner
Stratfor Worldview’s 2018 Annual Forecast
Stratfor Worldview’s Decade Forecast: 2015–2025
Subscribe to the Stratfor Podcast
iTunes - Google Play - Stitcher - SoundCloud - YouTube - RSS Feed
Have a question or comment for the Stratfor podcast? Leave us a message and we may include your comment in a future episode. You can leave a message for our podcast team at 1-512-744-4300 x 3917 or email us at podcast@stratfor.com.
Transcript
Emily Hawthorne [00:00:00] I'm Emily Hawthorne, a Middle East and North Africa Analyst at Stratfor, and this podcast is brought to you by Stratfor Worldview, our premier digital publication for objective, geopolitical intelligence and analyses. Individual team and enterprise memberships are available at worldview.stratfor.com/subscribe.
Mark Flemming-Williams [00:00:28] With the Euro zone being a monetary union, and all of the economic disruption and panic that we saw in 2015, it's a story that you have to keep an eye on, and so any tool which will help you understand that a little bit better, might be helpful for the business community.
Ben Sheen [00:00:54] Welcome to the Stratfor podcast, focused on geopolitics and world affairs from Stratfor.com. I'm your host, Ben Sheen. In this episode of the podcast, we discuss forecasting and, more specifically, the future of European integration. In a new collaboration between Stratfor analysts and Good Judgment Incorporated. As Startfor, we employ a very specific methodology, grounded in the practice of applied geopolitics to interpret the true significance of today's global events and develop a more accurate view of the future. The team at Good Judgment uses a quantitative approach and state of the art crowdsourcing to apply numeric probability estimates to their forecasts for discrete events. In part one of the podcast, we explore how these methods differ, as well as how qualitative insight and quantitative rigor can be applied in unison, to improve the forecasting process. We'll be talking with Stratfor's Rodger Baker, and Good Judgment's Philip Tetlock, along with Warren Hatch. Then, in part two, we'll see how we're putting this collaboration into practice, as we question the future of European integration. Thanks for joining us. And here with me on the podcast we have superforecaster Philip E. Tetlock, along with Dr. Warren Hatch, president of Good Judgment Inc. And in the studio, Rodger Baker, who's our VP of Strategic Analysis here at Stratfor. Rodger, Warren, Philip, thank you for joining us.
Dr. Philip E. Tetlock [00:02:18] Pleasure.
Warren Hatch [00:02:19] Hello!
Ben Sheen [00:02:20] One of the things that we all agree on is that forecasting is difficult. It's an extraordinarily convoluted process and actually our two groups, we come at it from slightly different angles, but we all seek to fundamentally try and forecast what will come next. What I'd like to start with is our different approaches to actually, the business of forecasting and seeing what will come next in global affairs. Rodger, would you like to start us off with the approach we take at Stratfor when it comes to forecasting?
Rodger Baker [00:02:47] Well, when we look at forecasting, I think a couple things that we have to think about in regards to which tools does one apply to forecast, to try to see the direction of the future. Because, some of that's going to be shaped on what's the time scale that we're looking at? What's the scope of the issues that we're trying to forecast? There may be some tools that are going to apply that are going to work much better for shorter timeframes or for more discrete types of questions, other tools that will work very well for longer timeframes and less defined questions. When we look at it, we start in some ways from a top-down approach, we use geopolitics to help us understand the shape of the world system, how the world interacts, and how it got to be where its at today. And from that, and from the study of the past, we try to understand the various factors that are intersecting at any given moment, that are working together integratively or pulling apart from each other, to shape the direction of the future. When we're building out forecasts, we're starting with the world model that we have, we're looking at the past as it led up to that, we're trying to identify both the direct elements that impact decision-making or changes in that trajectory in the past, as well as some of the indirect elements that play on that. Then, in some ways we'll linearly play it forward, knowing that linearity is certainly not the likely path, and from that we'll try to look at the constraints and the compulsions that may start to adjust those lines
Rodger Baker [00:04:23] as we lead forward. In some ways it's very much a top-down approach from the idea that a study of geopolitics will give us a working model of the world.
Dr. Philip E. Tetlock [00:04:34] That's a very clear and even elegant characterization of the forecasting process, and I don't think Good Judgment Inc. has a quarrel with anything you said. Good Judgment puts a lot of emphasis on the value of having a diversified portfolio of research tools, some more qualitative, some more quantitative, and I think Good Judgment Inc. is putting particular emphasis right now on bridging the qualitative-quantitative divide, which has historically existed in the forecasting community. On the one hand you have big picture scenario thinkers who tend to focus on long-term, open-ended sorts of outcomes, the future of the Euro zone, the geopolitical balance of East Asia. And on the other hand you've got these very micro, quantitative forecasters who focus on what's the price of oil going to be next month, will there be a violent incident here or there, what will be the range of the next North Korean ICBM test? So you've got these different approaches, and Good Judgment Inc. has been focusing, I think somewhat successfully recently, on developing techniques that link the micro, quantitative, narrowly defined forecasting outcomes, to these more global scenario types of outcomes. For example, you take a Davos style theme like fourth industrial revolution driven by strong artificial intelligence, will destabilize white collar labor markets by 2040 or 2050 or massively destabilize those labor markets. That's one of those big open-ended sorts of scenarios. It's very hard to wrap your head around it.
Dr. Philip E. Tetlock [00:06:12] On the other hand, you've got predictions about how many jobs in radiology or accounting will be lost to automation in the next year, or two years, or three years. Those are much granular, specific and quantitative. Now, what Good Judgment has been doing has been developing question clusters of micro-indicators that cumulatively can tip the scales of plausibility one way or the other about which scenario trajectory we're on, and do so more rapidly than was previously possible. For fourth industrial revolution, you would try to triangulate the big concept with a series of micro-indicators, like jobs in radiology, jobs in accounting, robotics spending at US exceeding 200 billion by 2018 or 2019, self-driving vehicles picking up passengers for hire in a major American city by the end of 2019. Some new triumph for one of the AlphaGo programs and so forth. Each of those things in isolation doesn't strongly tip the scales about whether we're on a historical trajectory towards a fourth industrial revolution, Davos style, but taken together, cumulatively, they do tip the scales somewhat.
Rodger Baker [00:07:24] Well I think that gets to one of the key points and key aspects of quality forecasting, is the idea that on the one hand, you can't oversimplify things down to the most narrowest, and on the other hand you can't allow things to go so broad and so wide that there really is no way to test what's going on. And some of that comes down to how we recruit, how we train for mindset, and how we really try to think about what are the subsets of the questions that we're asking. We use the traditional tools of intelligence to apply to these broader themes, and then go in and ask things. You know, if we're looking at self-driving cars and the quantity of them on the road, what would be some of the elements that we would need to see into that, what are the legislative and legal hurdles or opportunities? What's the physical infrastructure like? Is there a consistency in physical infrastructure that may allow this to accelerate in certain geographic locations but slow it in other geographic locations? Are there key nodes in the world where if it expands at a certain pace it's going to create a cost-effectiveness that's going to accelerate it in other parts of the world, because the large corporations are going to aim on it? Or are there places where it may concentrate but it will have minimal ripple impact from there?
Dr. Philip E. Tetlock [00:08:44] Very reasonable.
Ben Sheen [00:08:45] It sounds like to me, that there is indeed an overlap here and there's an opportunity to drive collaboration, which is really what we're looking to do between Stratfor and Good Judgment Inc., to try and work out how best to harness our approaches, to give the most accurate outcome.
Rodger Baker [00:08:59] Well I think there certainly is. Sometimes people see in forecasting a false dichotomy of quantitative and qualitative, that they're somehow completely different things, that they contradict each other or that they're from totally different schools or different approaches. It's hard to see, for example, a "qualitative forecast" that at some point doesn't have a lot of quantitative elements that go into it, and it's hard to see, in many ways, a quantitative forecast that didn't have certain qualitative and judgment calls made in assessing which types of data points would be put into it there as well.
Dr. Philip E. Tetlock [00:09:34] We're in danger of agreeing on almost everything.
Rodger Baker [00:09:37] Well I think this provides us an opportunity, though, as we're thinking about it, because I know that well, both of our organizations, really do see of reality in the idea of forecasting. There appears particularly in the realm of political risk, or that general space, a perception that maybe it's really not as viable, it's really not as good. It's based on some very narrow sense of predictions, people predicting, well, "There's no way Donald Trump would win," or, "There's now way Brexit would happen." And I think some of that comes instead from things that both of our organizations are looking at, which is: how do you look and tease out, or make forecasters aware of inherent biases and some of the traps of analysis and some of the traps that fit into trying to prove yourself right, rather than being willing to look for the things that falsify your theories.
Dr. Philip E. Tetlock [00:10:30] Right. Well, I'm glad you brought up the 2016 election, because it's an interesting object lesson, I think, for all in the political forecasting community. On the one hand, you had Sam Wang at Princeton with his poll aggregation techniques, putting the likelihood of the Hillary victory around 95%, with Nate Silver throttling back and putting it at only at about a 70% probability. Virtually all of these sophisticated poll aggregators thought it was more likely than not that Hilary would win the election. In the eyes of the public, they all look quite profoundly wrong, and it's a very hard sales job when Nate Silver appears on one of those comedy shows after the election, and he tries to explain why he thinks that a 70% probability of Hilary winning the election was probably roughly right. And that if you were to re-run history 100 times, Hilary would probably have won about 70, 70 of those. Donald Trump would have won about 30, and we wound up in the 30% likelihood world. People hear that and they roll their eyes and say, "What kind of metaphysical nonsense is this?" It's no wonder that a prudent forecaster wants to retreat to vague verbiage forecasting and say, "Well, I think it's a distinct possibility Trump could win," and if Trump wins, I say "Ha! Told you it was possible." And if he doesn't win, you say "Ah, I just said possible."
Ben Sheen [00:11:49] Something I'd like to zero in on here, is really expressing the value of forecasting for our audience. Because it seems almost people have gotten to a place, because of something like the the weather forecast, and go "Ah, they get it right some of the time, but it's a coin toss." Whereas even in meteorology there's a lot you look at that it helps you come up with your inductions and your conclusions, whereas actually in the realm that we're in, with the amount of data we have to crunch and sort of the specificity with which we look at some of the information and how we process it, how do we express this in a value that is actionable for our respective audiences?
Rodger Baker [00:12:21] From the Stratfor point of view, where we try to come at this is from the idea, I mean, it's from the name of the company, from strategic forecasting. It's in some ways trying to predict the flow of future history. What are the broader trends that are going to be shaping the space for decision-makers to act within? Almost more than trying to predict what is the specific choice they're going to make at any given moment, because that really does often come down to the vagueries of the individual. How well it plays out may be shaped by these much broader forces to a greater degree than the ultimate decision that they choose to make. And so we do try to come at it from that perspective but that also, will help people, particularly at a time when you have so much information coming, and so much information coming in isolated or narrow channels, where all of the information is presented as if it's all equal, and yet all of it is either contradictory or there just seems to be too much information there. Being able to have that world model and try to be able to tease out. Okay, it's easy to tease out what's unimportant versus what's important. Well-thought individuals can do that. But how do we find what's significant versus what's merely important, and at least ease us into that space? And if we can give people even a slightly higher chance of confidence of understanding the direction of the future, they can make better preparations. And I know I've slipped into some vague verbiage there,
Rodger Baker [00:13:48] but I'll let my cohorts on the other side of the line retort.
Dr. Philip E. Tetlock [00:13:53] I think it may have been Aristotle, may have been Einstein, I'm not sure who, but some very wise person in the past said something along the lines of "Seek precision in so far as the nature of the subject permits." We're in fundamental agreement on that from which to proceed. We don't really know ex-ante how useful it's going to be to try and quantify our probability judgements.The position of Good Judgment is, it's a good idea to try. And see what happens, and see what we can learn. It's interesting, if you visit the Federal Reserve for example, you find vast number of very skilled econometricians there, cranking out all sorts of forecasts and scenarios for the interdependencies among the economies. Lots of quantitative estimates, lots of probabilities, yet when you listen to the chairman of the Federal Reserve you are bombarded by vague verbiage, because obviously the Federal Reserve believes that the world is not ready for precise quantitative probability estimates. It hinges on the sophistication of the client. Some clients are ready for precise probability estimates and they make a good faith effort to try to combine the strengths of qualitative and quantitative. Other clients are going to be less intellectually mature and they're going to jump up and down and get really upset when Nate Silver says there's a 70% likelihood of Hilary winning and Hilary loses, they're going to say "Nate Silver doesn't know what the hell he's talking about." But Nate Silver, it turns out,
Dr. Philip E. Tetlock [00:15:21] is a pretty well-calibrated forecaster.
Warren Hatch [00:15:24] Just one thought to pick up on what was just said, is how to convey the importance of a numerical probability estimate, about an event such as we've been discussing. A forecast in isolation is not terribly meaningful. Where it begins to gain meaning is when you can see the track record of those who are making that probability estimate, and this is what the superforecasters at Good Judgment have also been very skilled at doing through a lot of experience, a lot of feedback, and having a track record. But anyone who's using a probability estimate about some consequential event, we can observe what their track record is. That's the key for it to gain meaning, is we observe the track record and see if the estimate they provide matches up with the reality that then follows.
Rodger Baker [00:16:11] I'll add onto that in saying that it's very important what effort method one uses in forecasting, that they're very open and honest in showing their past examples, in showing their success rate. It's something we've done from the start here at Stratfor, where we decided at the beginning, if we make mistakes we're going to admit to them, and when we make a forecast we're going to make a singular forecast, we're not going to give the three or four or five possibilities, we'll stick our neck out on the line on the one that we think is the most likely direction, and then be the first to admit when we're in error and use that to constantly improve our back end capability of making better and better forecasts as we go and so there's quite a bit of interest here in looking at all of these other tools that we can use to help improve various aspects of our forecasting process.
Dr. Philip E. Tetlock [00:16:59] In so far as Stratfor is willing to go down that path and as you say, stick it's neck out. In so far as Stratfor is doing that, I strongly commend you. A lot of organizations have a hard time doing that, there is a paradox here, that is what keeps you politically safe in many organizations is stick with vague verbiage that straddles both sides of maybe. Convenient phrases like "distinct possibility" that can be interpreted ex-post at your convenience. It keeps you politically safe, but it also makes it impossible for you to become well calibrated, it makes it impossible for you to do what former Chief Risk Officer at a major hedge fund once called "distinguishing 40/60 bets from 60/40 bets".
Rodger Baker [00:17:46] And the key for all of us is how do we continue to improve and make this much more useful for the consumer, that they're able to understand just what the forecasts telling them, and how to help to integrate that into their day to day decision-making—
Dr. Philip E. Tetlock [00:18:00] Right, and if you're talking to Goldman Sachs it's one thing, and if you're talking to certain other entities, it's another.
Ben Sheen [00:18:05] I'm actually really excited to talk further about some of the specific collaboration that we've done, looking at some specific events in the world. I guess that's all we've got time for for now, but I'd really like to say a big thank you to Philip and Warren for joining us in the podcast today, and also to Rodger for coming in from Statfor. Gentlemen, it's been an absolute pleasure, thank you very much for joining us.
Dr. Philip E. Tetlock [00:18:25] Thank you.
Rodger Baker [00:18:26] Thank you!
Ben Sheen [00:18:28] In the second part of the podcast, we go more in-depth on this new collaboration and how we're levering Stratfor's qualitative insight and Good Judgement's quantitative methodology to ask focused questions about the future of European integration. If you'd like to read Stratfor's standing forecasts on the future of the European Union, we'll include links to our annual and decade forecasts in the show notes. If you're not already a Stratfor Worldview member, you can learn more about individual, team, and enterprise access, at worldview.stratfor.com/subscribe. Now, to our conversation about the future of European integration, with Stratfor senior analyst Mark Flemming-Williams, and Good Judgment's Warren Hatch. Mark, Warren, thank you so much for joining me today on the podcast.
Mark Flemming-Williams [00:19:15] Thank you for having us.
Warren Hatch [00:19:16] Thank you very much.
Ben Sheen [00:19:17] Now, in our first conversation about the collaborative project we did between Good Judgment and Stratfor, Rodger Baker and Philip Tetlock talked a little bit about the different approaches to forecasting. What I'd like to talk about now is the actual project itself that we collaborated on and the ways in which we brought our different forecasting talents to bear on it. What exactly are we looking at in this joint project?
Warren Hatch [00:19:41] What we're launching is a brand new Future of Europe Index, and what it is is a distillation of currently 17 forecast questions about the economy, politics, and other news affecting European unity over the next year to 18 months. And what we've done is put that together into an index so that you can monitor at a glance whether the stresses are rising or falling according to the superforecasters on those sets of questions. Now, what's made it exciting to be working with Mark and your other colleagues at Stratfor is to compare notes about what are suitable questions to be forecasting to get at this bigger topic, cause that's tricky, you need to have good questions to have good forecasts, and I think we've ended up with a very nice, rich cross-section that people will find useful, interesting, intriguing.
Ben Sheen [00:20:40] And this is clearly a huge issue, because there's a massive focus on Europe at the moment. Mark, from your perspective, what did we really contribute to the forecasting aspect of this?
Mark Flemming-Williams [00:20:49] Good Judgment is attempting to find questions which will measure, specifically in this case, whether Europe is becoming more split up and that's what this fusion index is around. That's a question that we find very important as well, it's one of our key themes. Good Judgment came to us with some questions that they were already asking, that they'd put to their superforecasters, and already had ongoing questions. And we helped consider some of those question, because in what Good Judgment does, choosing the right question is key, whether it's going to give you a relevant answer to your overall thinking process. We tried to specify a little bit more about what it meant for Europe to be breaking up. For example, some of the initial questions were around whether Catalonia would secede from Spain and that would be a, obviously, a very bad thing for Spain, but there was a question about whether it would be a bad thing for Europe, because Catalonia quite possibly as an independent country might join Europe, and so the European Union might end up with more members and Catalonia might be an even more staunch member than Spain was. There could be some breaking up happening within Europe, but was it a breaking up of Europe? That was our first key concern, and then another one was more around slightly fiddly business around whether Russia were to become more aggressive on Europe's eastern border, and maybe in an extreme scenario if Russia were to invade somewhere like Estonia, which is a member of Europe,
Mark Flemming-Williams [00:22:29] then would that be seen as a breakup of Europe? And that was an interesting question, and we came back and were thinking it through, because actually what we have seen in terms of when Russia became more involved in Ukraine and there was the European sanctions as a result of Russian activity in Ukraine, we actually saw more European cohesion as a result of Russian aggression, it's one of the things which actually has been able to bring the rest of Europe together. Angela Merkel was able to corral European states in order to see off the Russian threat. Again, at first glance, it looks like it is disintegration of Europe, but when we looked a little bit deeper, then perhaps that wasn't such a fitting question for the broader. We provided a little bit more context and a little bit more help in choosing the questions which would help answer the overall questions of "Is Europe facing some kind of disintegration?"
Warren Hatch [00:23:28] Maybe I can just amplify a little bit on what Mark just said, 'cause I think Catalonia in particular is a really nice case study for our different approaches, and how they can be usefully combined as we've done here. And so the way Mark set that up with Catalonia, that well, if Catalonia was successful in seceding, would that lead to greater European integration or less? And from our point of view, that's a different question, really. Question number one, does Catalonia secede? Does it have an independence referendum? That's really what we were asking at the time. At the time, it's a safe proposition that most commentators saw that as a negative with respect to European unity. That was just one question in the index, we may have different views, hopefully, across the index as a whole. It balances out enough that the signal is still reliable even if we may have a different interpretation of one or two questions. But in the case of Catalonia, then, that's part one. Part two is what it means from there. It's a conditional question on our part so it would have been interesting to have posed a question that got to European unity if Catalonia had an independence referendum versus if Catalonia did not. And from a super-forecaster point of view, what you can do is observe the spread. You can see whether Catalonia's independence referendum would be impactful from a forecasting point of view on that other specific event. Now we didn't do that in that case, but it's the sort of thing that we want to be building into the index,
Warren Hatch [00:25:13] hopefully with Stratfor's input for future questions as well.
Ben Sheen [00:25:18] This seems like a huge amount of data to actually track and comprehend, and then ultimately share, because it seems to be that asking the right questions is part of it, but then also you have myriad responses that are all going to flow into each other. How do we actually make sense of all this data and how do we actually use it to make meaningful deductions about the future.
Warren Hatch [00:25:39] The advantage of the index is that you can summarize a lot of information at a glance and then drill down into it to get whatever level of detail interests you, so if you're just concerned about whether Europe's near-term outlook is looking a bit more favorable to unity or less favorable to unity, because you've got portfolio positions or you're making a business investment or you just need to know for whatever purpose you might have, you can use this as a straightforward simple indicator to let you know, can you sleep soundly at night? You can look at it and say "Oh okay, well things seem to be kind of going along. There are all these scary headlines but it really doesn't matter too much on the near-term outlook." Or alternatively, "Oh! There's something subtle that may have emerged and I wasn't focused on it. This will help me find what I need to focus on to drill into it." What you can do is see the index, it goes, say up significantly, okay, well what's driving that? You can go in and see what the individual questions are that are helping support greater unity. Or the other way around.
Ben Sheen [00:26:53] If you have slightly different approaches to forecasting and the way we tackle certain issues, was there anything where it came to collaborating that supported either of you, Mark or Warren, in some of the differing approaches of the way you normally do business.
Mark Flemming-Williams [00:27:07] We've got very different approaches, it's just a different way of approaching the world. Because we very much have an ongoing model and we have forecasts which run... We've got the decade forecast, and then we've got the annual forecast, then we've got the quarterly forecast, so we've got a picture of, and established since the foundation of Stratfor, view of the world, which we are constantly updating and keeping going, and tweaking and playing into as we go along. It's a different approach, whereas Good Judgment is very much trying to find specific questions which will give the answer. We are, instead, looking at the whole world and extracting the answers from the overall model. It's a different way of going about the same problem.
Warren Hatch [00:27:58] And where it comes together, is we're very agnostic about the world. Though we're very quantitative about trying to understand specific events, and where it can come together is, Stratfor does have a model of the world, this generates a lot of good questions. And a lot of these question can be forecast. And it can also be a source of well, if this model is unfolding in this particular way, there's certain things we should expect to see consistent with that model, and that's where we can create forecast questions. And we can also add in questions that maybe question whether that model is unfolding in that way or not. And then we can generate an interesting, rich mix or forecast questions that let us monitor and see whether the world is moving in that direction at the moment or not, now just because it may be moving in that direction over the next year or so doesn't guarantee by any means that that's the way it's moving over the next decade, and that's where skillful analysts certainly come in to connect those dots and help tell that richer story, that combines what may be happening in the near-term, which is the forecasting horizon of say the next 3 months to 2 years where super-forecasters excel, with the longer-term perspective that Stratfor brings that can go well beyond that.
Ben Sheen [00:29:27] It's a really powerful mix because everyone we encounter, everyone has a, shall I say more than a passing interest in what will happen next, what will play out in the future. One of the things I know we both focus on is making this information actionable. What is the real value of this combined effort, for example the business sector, if you work in say, manufacturing or finance. How can you actually use these forecasts to help you with your long-term business projections.
Warren Hatch [00:29:55] Well, one way, is say you're a portfolio manager with a large currency exposure in Europe, and in your view, a stronger European Union would be impactful on the value of your Euro position. This is a way to monitor how concerned you should be about your position. It is also a way to identify potential early opportunities if the trends begin to move in a different direction. It's also a way, if you're for instance making a decision to invest in Europe, is this a good time to do that or not? And help you focus on the things you need to focus on.
Ben Sheen [00:30:36] Mark, anything from your perspective?
Mark Flemming-Williams [00:30:39] As Warren says, Europe is obviously one of the three big economic hubs in the world at the moment. We've seen crises in the last, which is a lot of crises, in the last five to 10 years, and they strongly affect the markets and anyone who is considering making investments in Europe, or considering holding the Euro, or holding any other currency to be honest, because the Euro is such an important part of the currency market. It's all about whether, with the Euro zone being a monetary union, and all the disruption that can come from even the smallest, one of the smallest players like Greece, leaving. All of the economic disruption and panic that we saw in 2015 is the story that you have to keep an eye on and so any tool which will help you understand that a little bit better, and gain more insight into answering that question, I think can't but be helpful for the business community.
Ben Sheen [00:31:35] Well, that's actually an excellent place to wrap up, because thanks to this collaboration we've offered an important new tool to actually help not only businesses, but inform individuals, and even governments could use, perhaps, some of the future trajectory of Europe itself. How do people get involved in forecasting at a grassroots level, if people wanted to contribute to the conversation, or perhaps share predictions they made of their own, is there an avenue to do this in any way?
Warren Hatch [00:32:01] Absolutely. Here are a few that would actually be really interesting for people who are interested in developing their own forecasting skills. One is, I believe that we're going to be having a forum discussion with Stratfor analysts and some superforecasters to talk about forecasting, tackle some specific questions, and generally compare notes. Another way is to go to the Good Judgment Open, which is our public forecasting site, try their hand, there's a rich mix of questions there. You can make your forecast, once they close you will get feedbacks, you will get scores. It's also the funnel for our future superforecasters. In fact, just this last few months we brought in a class of 2018 superforecasters to our professional ranks. And a third thing is brand new. If you missed a chance to be a superforecaster in the original research project, a new one is just getting underway, sponsored by IARPA, called the Hybrid Forecasting Competition. Recruitment is just underway now, it's going to be a big multi-year project, really exciting, getting at how you can combine the best of humans and machines to push the accuracy of forecasting even more.
Ben Sheen [00:33:19] Fantastic. That sounds like an amazing opportunity and clearly something we'd be interested in participating in. Mark, Warren, thank you so much for joining me today.
Warren Hatch [00:33:27] Thank you very much. Appreciate the time.
Mark Flemming-Williams [00:33:30] Thanks very much.
Ben Sheen [00:33:43] And that concludes this episode of the Stratfor podcast. Watch for an update on Stratfor's collaboration with Good Judgment Inc. on our website, worldview.stratfor.com. We'll include links in the show notes to a pair of columns we published on the topic of qualitative and quantitative analysis. The Numeric Problem was penned by Stratfor Vice President for Strategic Analysis, Rodger Baker. The Numeric Solution was written by Dan Gardner, who co-wrote Superforecasting with Philip Tetlock, from part one of the podcast. You can also find a complete interactive transcript of this conversation on our podcast page. That's at worldview.stratfor.com/media/podcasts. Worldview members can also contribute to this conversation and engage with Stratfor's analysts, editors, and contributors in our members only forum. If you have a comment or an idea for a future episode of the podcast, email us at podcast@stratfor.com or give us a call on 1-512-744-4300 extension 3917 to leave us a message. We really appreciate your feedback. And for more geopolitical intelligence, analysis and forecasting that reveal the underlying significance and future implications of emerging world events, follow us in twitter @Stratfor.
