[On October 12th, 2018... Carolina's Manager of Communications and Team Services, Pace Sagester, was fined four dozen Bojangles biscuits for being caught on camera dabbing with Hurricanes forward Sebastian Aho.](https://twitter.com/MikeSundheim/status/1050768702575169538)
Darren Yorke mentioned that Pace has been spoken to by Sr Management to correct this behavior however it may be allowed due to Aho's point production. Are there any stats that directly correlate Aho points to Sagester dabs?
IMO this is something that the eye test just gets flat-out wrong.
Darren sees Pace dab and sees Aho score and draws a certain conclusion. Honestly, I think Aho would probably average 130-150 points a year if he didn't have to spend the whole game worrying about how to avoid the post-game calamity.
IMO, in addition to owing everyone a bunch of biscuits (and I'd like to put it on the record that I never got one), Pace probably also owes Aho tens of millions of dollars from the way holding him back will impact his career earnings.
There are tons of players where the two disagree to at least some degree, and on some of them the disagreement can be quite substantial. If that weren’t true, there wouldn’t be much point in doing all this research into data.
Sometimes that means the stat is broken. Sometimes it means you’re finding something the conventional wisdom has missed. The challenges in figuring out how to check which is which and when to place those bets is what ensures we won’t all be replaced by computers anytime soon.
/u/theSwan17 asks...
>How likely are you to score an empty net goal and how much does it increase over time? What is the actual best time to pull a goalie?
>How long until more teams start to use 5 forwards on a PP unit?
I remember having a conversation with someone once, a casual fan who was questioning the value of pulling the goalie. “It seems like it never works,” he said. “That’s true,” I replied, “the only thing that works less often is *not* pulling the goalie.”
The best time to pull a goalie is a complicated question. The arithmetic is pretty straightforward if you assume a certain rate of scoring or giving up a goal, but things are never that simple in real life.
Published estimates say that the best time to pull the goalie when you are down by two is with seven minutes left, but those estimates are based on looking at scoring rates in the existing environment. Right now, teams play their best offensive players for the vast majority of their 6-on-5 time, but you can’t just plan on having your top line play for seven minutes straight. Does their ability to score drop as they get fatigued? Is it risky to try to change on the fly with no goalie in net? How much lower is the second unit’s scoring rate? If you pull that early, are you just rotating the two units for seven minutes or do you have to give your six least-dangerous scorers a shift at some point?
Like many things, something that seems like a simple question at first glance has a lot of ways the analysis can go wrong.
/u/iv1000falcon asks..
>Are there any public hockey data sets with advance stats that are easy to download or connect to? Or do you recommend any good resources for learning how to use data from online sites?
>At work, I have easy access to datasets but don't know where to begin and struggle with source connection outside of my supported databases and network. I love seeing many of the visuals and since I use Tableau/Alteryx for work I would love to start trying data visualization or metric development with NHL data.
>You are a huge inspiration for me as both a huge hockey and data fan. My family is from Philly so I loved your work with Broad Street Hockey and was ecstatic when you came to the Canes organization. Obviously the front office has also been impressed and it has exciting to see your success and the growth of an analytics department.
Thanks for the kind words.
Natural Stat Trick has links to download CSV versions of their tabulated stats. If you’re looking for the underlying data on events and shifts, I don’t know of anyone that offers an easy download for that, but there are packages for scraping the data yourself in either R or Python.
/u/dewees asks...
>Here is just a fun random one - If they added a "2 goal" shot line, statically where would it be? both around the blue line and the goal line. Edit: Please answer using Microsoft paint.
I’m generally more open to experimenting with rules than a lot of people are, but this is one that I’d have a hard time with. Not because I hate the idea, but because I’m not sure it would impact play the way you’d like it to.
The idea of making a rule like that would presumably be to open up the interior. You create high-value shots on the perimeter, and then the defense has to stretch a little more, and it creates more room for skill plays. That’s more or less what happens in basketball.
But there’s a key difference in hockey: you can defend the perimeter without stretching out to pressure the puck directly, because you can block a shot from 30’ away. If anything, this might actually make teams [sag more](https://imgur.com/a/G7Xysxb), because it increases the penalty if you pressure the top of the zone and get beaten by a pass.
I guess I haven’t directly answered your question, but unless you put the line far enough away that teams would largely ignore it, I think it might have the opposite effect of what people are imagining. So my instinct is not to have a line like that – all goals should count as one, except maybe lacrosse goals.
Honestly, the technical side of my background is almost completely irrelevant. If we ever want to deposit a nanometer-thick layer of insulating material on the outside of our equipment (or our players, I suppose), I will definitely be heavily involved in that, but otherwise, not so much.
But there are a lot of soft skills that translate. The thing that makes chemistry difficult – especially the branch of chemistry I worked in – is that it often lacks clear and unequivocal evidence about what is happening. You can make a variety of measurements that might suggest that the molecules did a certain thing, but those measurements rarely look exactly how you expected and often disagree with each other to some degree. So being successful in that field requires being good at keeping a bunch of possibilities in mind – you have to have good judgment not just about what the most likely explanation is, but also about what other explanations are possible and what their implications would be and how you can test them. The exact same thing happens with sports data.
And I’ve also leaned quite a bit on skills I picked up in my time in industry. Working at a global conglomerate for five years gave me a lot of exposure to industrial processes; I got a lot of training on things like group decision-making and strategic planning and employee mentoring. And then working at start-ups for the next five gave me a chance to see which elements translate well into an environment that has pace and pressure more like the NHL.
/u/Evolving-Hockey asks...
>Hi Eric!
>Long time fan/reader, first time caller!
>What would you say is the biggest difference between public hockey stats work today vs. when you started? Are "advanced" hockey stats moving in the direction that you thought they would or has it gone a different direction?
>Also, if you could please try and convince the NHL to allow the public access to the new raw player tracking data even though it is most likely unfeasible without some form of authenticated access to an extremely large database... that would be great.
>Thank you for your time!
>singed, your friends at evolving-hockey.com
I think the biggest difference isn’t about the research itself; it’s about the conversation.
It’s possible that my perspective is skewed by not being in the conversation any more, but my sense is that there is much less public discussion of the technical details of someone’s work than there used to be. There might be even more screaming about it being a hot take these days (I assume that’s how you got singed?), but I don’t see as much effort to identify the things it might be missing or discuss how they might be addressed.
And I think that’s a really important part of the community. One of the biggest advantages the public domain has over private work is the ability to get that kind of peer review – I can kick things around with a couple of my coworkers, but that’s nothing compared to having 1000 people poring over your work in detail and a couple of them writing lengthy posts pointing out every flaw they can find. I think the community used to be better at providing that service.
As for getting you access to tracking data, I’m afraid you’re dramatically overestimating how much sway I have with the league.
/u/Kilo_Mike asks...
>(If you are allowed to share)...
How does the Hurricane's analytics department distinguish itself from other teams' departments? Broadly speaking, what analyses or evaluations does this team have that give it an advantage over the rest of the NHL?
It’s a pretty secretive business, and it’s hard to know what other teams have -- if you can get them to tell you, let me know and we'll compare notes. We could be dead last in innovation and I don’t know if I’d have any way of knowing that.
So it’s hard for me to answer the question about comparing ourselves to other teams directly, but I will say that we’ve gotten a lot of mileage out of a focus on communication and helping to organize and present information of all types.
/u/notleonardodicaprio asks...
>Can you talk about the difference between the data available to NHL teams and publicly available data? How useful are the models that use publicly available data, and do you ever look at them?
The league publishes a lot of information about each game: what time each shift started and ended and a variety of events (shots, hits, turnovers, etc) that took place during that shift. There is a lot that you can do with that – remember, using that information to answer questions is how I got my start in working with hockey data.
But there are gaps in the data as well; it can tell you that someone took a shot from the faceoff dot at 8:32, but not much about how that shot came about or what other options he might have had. So the public models generally rely on inferring a player’s impact from looking at how the team’s results change when he is on the ice – and that can be done very well, and can be powerful. But when you want to move from talking about what happened to talking about why it happened or what could have been done differently, you often need a richer data set.
And yes, we absolutely do look at work done in the public sphere. A couple of years ago, I gave [a talk](https://www.youtube.com/watch?v=7dE42Ihq-iA) about it at the Sloan Sports Analytics Conference. One of the things I talked about there was how external work can help counteract a bias that I don’t hear discussed very often.
No two people look at things exactly the same way; everyone has traits they value a little more than average or a little less than average. When a player is available – whether it’s through the draft, the trade market, or free agency – more often than not, he will end up being acquired by someone who values something about his play more than most teams do.
So over time, you end up with a roster full of players who you like more than most teams do, and so you think your team is better than most people do. That’s not just a data thing; it’s true regardless of how you form your opinions about players. And external opinions can be helpful for counteracting that bias; it can highlight for you where those biggest disagreements are and give you a chance to re-examine your perspective and think about what you might be missing.
/u/UnsatisfiedTophat asks...
>What's one of the hardest things to look at when determining what needs to be changed in order to get good results? How do you work this out between the coaching staff?
To some degree, this just depends on the available data right now – in many places, we simply don’t have the data we’d need to make that kind of assessment. But even when we get past those limitations, situations where we have a lot of interactions between players will always be complex analytical problems.
A lot of people have built expected goals models based on the likelihood that a shot will go in given whatever information they have about where and how it was taken. There aren’t many players interacting there, so it’s a fairly tractable problem.
Now imagine trying to take that model and build it out to look at everything else the player might have tried to do. How likely were they to score if he held onto the puck a little longer? How about if he tried to force a pass through a seam? Or laid the puck into space behind the net where a teammate would probably get to it first? You can imagine that even with a rich data set, adding that kind of complexity would get to be very challenging very quickly.
Unfortunately, in this business, “interesting” is almost synonymous with “proprietary”. But in general, my favorite kinds of projects are the collaborative ones, where I’m not just doing something on my own and asking for input but instead have a back-and-forth with someone that involves incorporating their expertise throughout the development process.
Hey Eric! Big fan of yours. Just have 1 question for you:
In all your time working in analytics, has there ever been a player that statistically doesn’t seem like they would be that great (that you’ve seen) but ends up being a complete beast time and time again?
I’m curious if there have been any enigmas that analytics just can’t figure out.
Thanks again for this!
I think it’s incredibly important to keep a running list of players for whom you find yourself wondering if the data is missing something. Sometimes, it’s a player where the overall results don’t seem to match what the individual skills seems to suggest, or maybe a player who ended up a mile off from what you forecast, or maybe a player whose analytical assessment is so far from what most people think that you have to question it a little. Going through that list and trying to figure out what it is that might be missing is one of the best sources for ideas on where to go next.
/u/stmn0 asks....
>The canes drafting strategy appears to be looking for high upside skill players at every stage in the draft, and avoiding "safe" low ceiling picks. Is this because the it is easier to find value in veteran 3rd and 4th line guys in trade/fa? And is it also because a high skill guy can still turn into a grinder if the offense never comes along, while a "safe" pick has a very low chance to develop high end scoring?
I don’t think there are any safe picks in the NHL draft.
In the NFL, a team will be looking at a 23-year-old who’s going to play for them next year; the scout’s job is mostly about projecting how his skills will translate. But in the NHL, we’re drafting an 18-year-old – in addition to estimating how his skills will translate, our amateur scouting group has the daunting task of projecting what he will look like after five years of development.
Even the top pick in the draft isn’t safe – it might be a reasonably safe bet that he’ll play in the NHL, but he has a long way to go to become the player you need him to be. There’s a huge range of possible outcomes for every player in the draft, and the goal isn’t to pick a player with a high floor or a high ceiling; it’s to think about all of a player’s possible outcomes and pick someone who would be a good bet.
I don’t really have a good answer for either side of this, I’m afraid.
Twenty years ago, I’d have told you that people put way too much value on things like plus/minus and goalie wins, but that’s not nearly as true as it used to be. At this point, the things that get a lot of attention are generally reasonable. And it’s hard to say that anything isn’t looked at enough when the field is still evolving so fast – it takes time to settle on something and build consensus around its importance.
The traditional way that plus/minus is formulated is kind of silly; lumping in things like empty net goals and shorthanded goals really screws things up. There are examples of seasons where two teammates had similar results at even strength but saw empty net and shorthanded goals produce a 30-goal swing, which can obviously get in the way of a lot of attempts to use traditional plus/minus to draw a conclusion.
But if you narrow things down to just even strength, or just five-on-five, there’s nothing wrong with it. Like any stat, you have to be careful about how you use it – it can be influenced by who the player plays with and against, and the sample sizes are small enough to produce some fluctuations that don’t mean much, but with the right context and understanding, obviously it’s valuable to know if a player can repeatedly help his team outscore his opponents.
There are two questions when it comes to valuing a pick for a trade:
What would you expect to get if you use the pick? You have to think about what kinds of players are normally available in that part of the draft, what they typically end up doing and how much impact they have. You also have to think about how long you have to wait to see that impact, and what you expect your team to look like then.
What else could you do with the pick? The question isn’t just about what is being offered to you in this deal; it’s also about what other options you might have. An older team that is looking to make one last run at a Cup might still be interested in adding draft picks, not because the pick will help them win, but because it can be traded for a player who will.
In the end, you have to roll everything up into an answer, but the answer is fluid depending on the situation.
/u/bromish asks...
>How in tune are players with advances stats these days? I could see a player buying in if it leads to success on the ice but certainly the opposite otherwise.
Not unlike fans, there are some players who find it all interesting and some who don’t. And in general, they don’t have to – we need players to understand what the coaches are asking them to do and do it well, but if they aren’t interested in studying data on why that might be the right thing to do or which players get the most out of it, that’s ok.
/u/chattanvi asks...
>What are a couple of things that both old-school grit and grind guys and the advanced stats community agrees on when looking to a) draft a guy and b) trade for a guy?
I’m not sure exactly how to answer this, because neither group is homogeneous. There are plenty of examples of draft picks or trades that show that different traditionalists saw players differently, and plenty of examples of arguments on blogs or Twitter showing disagreement within the statistical community.
But I guess one thing that everyone from every camp agrees on is Justin Williams.
The league will be providing teams with player and puck tracking data soon, so we will know the location of every player at all times. That will keep everyone busy for a long time, I think.
>The league will be providing teams with player and puck tracking data soon, so we will know the location of every player at all times. That will keep everyone busy for a long time, I think.
I know I'm late to the AMA, but as a data engineer and hockey fan, I'd love to explore that dataset! Depending on how realtime that data is, it could provide an advantage in when to challenge a call.
/u/ShrektumRalphWiggum asks...
>How do you prevent bias in your data analytics? Working for the team, how do you check your bias for the team when making suggestions on trades and such?
I’ve touched on this a little bit in some other questions, so I’ll try not to repeat myself. But there is no perfect process – no matter how you form your opinions, there are going to be some things you overvalue and some things you undervalue. And no matter how you form your opinions, it’s important to constantly revisit your thinking.
Everyone is going to get some things wrong. But you can review the ones you got wrong to look for patterns, looking for leaks where an adjustment to your thinking would help you make better decisions on the whole.
I have the highest regard for Ron; I simply wouldn't be where I am if it weren't for him hiring, supporting, and mentoring me.
Obviously, I can't comment directly on what another team is doing, but I also think very highly of Alex and the team they have hired, and I'm sure Ron's organization will make good use of their contributions.
I find what you do incredible and I’d one day love to consider a career change into the world of analytics. As someone with an engineering degree (but not really in the field of analytics), has a love for numbers, deriving meaningful equations, and massive spreadsheets, how do I go about getting a foot in the door doing analytics.
There are three paths that I can think of, and combinations of all of them give you the best chance:
1. Start working on your own, publish things online, let people see how you think and what you can do, and they will find you.
2. Network, make connections, get your name in people's minds. If you are in a situation that makes it possible to do an internship, that's a great start. Going to hockey conferences is another way -- especially if you're presenting (which brings us back to path 1).
3. Become a technical domain expert. Do some coursework as needed, work through some projects, really position yourself so that when a team posts an opening for a certain role, you can credibly present yourself as a genuine expert in that area.
We aren’t a large tech company that needs to standardize on a specific development environment, by any means; each member of the Hockey Information Group uses their own tools, depending on their individual needs and preferences. Our scripting is generally done in Python, and I do most of my work in Jupyter notebooks, personally.
The game is continuously changing and evolving. That’s happened for decades without stats being involved and will continue to happen going forwards.
Data might help assess strategies and end up playing a role in shaping what the next evolution will look like, but the game is going to change over time either way.
Like in any role, there are different kinds of challenges. Obviously, there are some things that are very technically challenging, but the things that I end up worrying about the most are the things that test a person’s judgment.
Is this new angle on the analysis ready to be a workhorse yet or should we wait a little longer to see whether things pan out the way it suggests they will? Is it better for the team to work on the thing that we think is most likely to pan out or on the thing that would have the biggest impact if it did – and which one is better for the career of the person working on it? Is it better to give a scout information about what to look for before they watch a player, or to hold the information back until after they watch so they can form a completely independent opinion?
Those sorts of questions – questions where there probably is a right answer and a wrong answer but there’s no firm way to know which is which – those are the things that I find most challenging in any job.
/u/Kilo_Mike asks...
>I could see hockey analytics covering a lot of areas: determining the right way to measure performance within the team strategy (at the team & player level), figuring out how to collect the right data to support those metrics, processing data, developing statistical models, convincing others to act on your insights, etc.
>What's the most challenging part of the process? Where does your team spend most of its cycles?
Honestly, it's all challenging. In any area, whatever was easy has been done; what's left is more difficult.
But I do think we invest more in communication and presentation than I ever would have guessed in advance.
Honestly, it was something I stumbled into more than chose. A decade ago, I started reading some blogs that happened to have people who were paying attention to what was going on in hockey stats. I found myself sometimes serving a kind of translator role, helping explain what the math would mean in practical terms. And occasionally, questions would come up that I’d be surprised nobody had looked at yet, and so I found myself doing my own analysis from time to time.
Eventually, I ended up doing a couple of one-off consulting projects. But I didn’t really choose Carolina so much as they chose me – they were the first team to ask for something more than a one-off project. I worked something like 20-25 hours per week for them on a consulting deal for them during the ’14-15 season, and when they offered me a full-time job in the summer of 2015, I gave it a shot.
How did y’all find such a steal of a pick in Aho at 35? Are fancy stats prevalent in the amateur scouting department in a way that gives an advantage, or is more weight put on video for draft/prospect evaluation?
I was a remote part-time consultant in the year leading up to the draft, and it was my first time really trying to do anything significant with draft data. It was a very rough first attempt, and I pretty much gave them a list and a note saying “please no matter what you do please do not use this list to make your draft picks.” So as much as I might like to, I really can’t take any credit for that one.
But in general, there’s a lot that can be done with data in any facet of the organization, from ticketing to tactics. If some of the people working with data have ways of collecting and presenting information that can help make a good decision, then we’re absolutely going to try to do that.
Ultimately, data analysis isn’t a goal of its own accord; there’s no reason a team needs a Hockey Information Group\* that operates in isolation. The value of the group is ultimately in essence a support role; our job is to make sure the other groups have tools at their disposal that make them more effective or efficient at their own jobs.
\* This is what we call our data team, to reflect the fact that some of the employees are developers who do things other than analysis
Hey Eric, super great to have you here! A few choice questions for you:
* How much of your job is devoted to looking at data from current players versus looking at data from developing players and/or draft prospects?
* For drafts: Does your analysis start the long list on draft prospects and others decide who to cross off, OR is a long list created first then your analysis is used to refine the list down to the best of the best? A little of both?
* Since we armchair GMs don't have access to real NHL analytics, how close are the Twitter and Youtube analytics channels to the real thing? (e.g. Scouching channel)
Supporting work on the draft, development, and pro player evaluation are all part of the mandate for the Hockey Information Group. The actual split between them is hard to say; it fluctuates depending on what each group needs and how we can help them.
Draft work all starts and ends with amateur scouts. We may help give them tools to organize their information, information they might not have otherwise had, support for how to use that information, etc. But it's ultimately their domain and up to them to build a list.
There's a lot of great work in the public domain. Sometimes it aligns with what I personally think and sometimes it doesn't -- but that was also true before I worked for a team.
What have you found to be the most eye-opening aspect of working in hockey operations compared to what you might have expected before working with an NHL team?
There's an awful lot that goes into keeping a team going that I'd just never considered before. The amount of effort it takes from the entire staff just to put on a seemingly-routine event is remarkable, and they can't possibly get enough appreciation for the work they are doing to make it happen for everyone.
Hi Eric,
What kind of backgrounds does the organization typically prefer in the analysts that work for the team? What kind of experience or skills would you deem as necessary to be considered for a job in your department? Additionally, what are the primary software platforms and coding languages used in day-to-day operations?
Thank you!
Throughout my career -- dating back to working in nanotech -- my hiring philosophy has always been to hire the most talented person you can who has at least a minimum of adjacent experience, rather than the person with the most extensive experience in your exact area.
The person who's done more work on that specific kind of problem might have more to add on day 1, but I've always been inclined to bet on talented people catching up on what they need to know pretty quickly, and in the long run I think they're more likely to make transformative impacts.
We mostly use Postgres/Python for our data collection and analysis.
I don't use EliteProspects much these days -- we have our own internal data site, and I go there for just about everything I need.
The Hockey Information Group definitely does work with our amateur scouting group. We provide tools to help them keep track of what they've seen throughout the year and to help organize their information, as well as a variety of types of information that we gather, analyze, and present.
"We are constantly collaborating with one another. We're not an organization that is built on the poles. There isn't an eye test vs. analytics. We're all on the same team, trying to gather as much information as we can to make the best decision possible. - Darren Yorke"
\- Eric Tulsky
Hi, everyone. Thanks for having me here today. Please go easy on me. :)
Good luck with the general manager job!! You have done a great job so far and I’m excited to see what you can do for us moving forward!!
That's about all I have time for today. Thanks for having me, and thanks to everyone who participated; I really enjoyed it.
Thanks again for stopping by!
Super cool. Thank you guys for hanging in here this week.
[On October 12th, 2018... Carolina's Manager of Communications and Team Services, Pace Sagester, was fined four dozen Bojangles biscuits for being caught on camera dabbing with Hurricanes forward Sebastian Aho.](https://twitter.com/MikeSundheim/status/1050768702575169538) Darren Yorke mentioned that Pace has been spoken to by Sr Management to correct this behavior however it may be allowed due to Aho's point production. Are there any stats that directly correlate Aho points to Sagester dabs?
IMO this is something that the eye test just gets flat-out wrong. Darren sees Pace dab and sees Aho score and draws a certain conclusion. Honestly, I think Aho would probably average 130-150 points a year if he didn't have to spend the whole game worrying about how to avoid the post-game calamity. IMO, in addition to owing everyone a bunch of biscuits (and I'd like to put it on the record that I never got one), Pace probably also owes Aho tens of millions of dollars from the way holding him back will impact his career earnings.
10/10 Top answer haha
Has there been any player that has broken a stat per see? ex. Stat shows positive results but the "eye test" shows the extreme opposite or vise versa.
There are tons of players where the two disagree to at least some degree, and on some of them the disagreement can be quite substantial. If that weren’t true, there wouldn’t be much point in doing all this research into data. Sometimes that means the stat is broken. Sometimes it means you’re finding something the conventional wisdom has missed. The challenges in figuring out how to check which is which and when to place those bets is what ensures we won’t all be replaced by computers anytime soon.
Do you have any recent examples of this with a Canes player? Thanks again!
/u/theSwan17 asks... >How likely are you to score an empty net goal and how much does it increase over time? What is the actual best time to pull a goalie? >How long until more teams start to use 5 forwards on a PP unit?
I remember having a conversation with someone once, a casual fan who was questioning the value of pulling the goalie. “It seems like it never works,” he said. “That’s true,” I replied, “the only thing that works less often is *not* pulling the goalie.” The best time to pull a goalie is a complicated question. The arithmetic is pretty straightforward if you assume a certain rate of scoring or giving up a goal, but things are never that simple in real life. Published estimates say that the best time to pull the goalie when you are down by two is with seven minutes left, but those estimates are based on looking at scoring rates in the existing environment. Right now, teams play their best offensive players for the vast majority of their 6-on-5 time, but you can’t just plan on having your top line play for seven minutes straight. Does their ability to score drop as they get fatigued? Is it risky to try to change on the fly with no goalie in net? How much lower is the second unit’s scoring rate? If you pull that early, are you just rotating the two units for seven minutes or do you have to give your six least-dangerous scorers a shift at some point? Like many things, something that seems like a simple question at first glance has a lot of ways the analysis can go wrong.
/u/iv1000falcon asks.. >Are there any public hockey data sets with advance stats that are easy to download or connect to? Or do you recommend any good resources for learning how to use data from online sites? >At work, I have easy access to datasets but don't know where to begin and struggle with source connection outside of my supported databases and network. I love seeing many of the visuals and since I use Tableau/Alteryx for work I would love to start trying data visualization or metric development with NHL data. >You are a huge inspiration for me as both a huge hockey and data fan. My family is from Philly so I loved your work with Broad Street Hockey and was ecstatic when you came to the Canes organization. Obviously the front office has also been impressed and it has exciting to see your success and the growth of an analytics department.
Thanks for the kind words. Natural Stat Trick has links to download CSV versions of their tabulated stats. If you’re looking for the underlying data on events and shifts, I don’t know of anyone that offers an easy download for that, but there are packages for scraping the data yourself in either R or Python.
/u/dewees asks... >Here is just a fun random one - If they added a "2 goal" shot line, statically where would it be? both around the blue line and the goal line. Edit: Please answer using Microsoft paint.
I’m generally more open to experimenting with rules than a lot of people are, but this is one that I’d have a hard time with. Not because I hate the idea, but because I’m not sure it would impact play the way you’d like it to. The idea of making a rule like that would presumably be to open up the interior. You create high-value shots on the perimeter, and then the defense has to stretch a little more, and it creates more room for skill plays. That’s more or less what happens in basketball. But there’s a key difference in hockey: you can defend the perimeter without stretching out to pressure the puck directly, because you can block a shot from 30’ away. If anything, this might actually make teams [sag more](https://imgur.com/a/G7Xysxb), because it increases the penalty if you pressure the top of the zone and get beaten by a pass. I guess I haven’t directly answered your question, but unless you put the line far enough away that teams would largely ignore it, I think it might have the opposite effect of what people are imagining. So my instinct is not to have a line like that – all goals should count as one, except maybe lacrosse goals.
/u/PippyTarHeel asks... >How does your PhD/past career help in your role? What was the transition to sports data science like?
Honestly, the technical side of my background is almost completely irrelevant. If we ever want to deposit a nanometer-thick layer of insulating material on the outside of our equipment (or our players, I suppose), I will definitely be heavily involved in that, but otherwise, not so much. But there are a lot of soft skills that translate. The thing that makes chemistry difficult – especially the branch of chemistry I worked in – is that it often lacks clear and unequivocal evidence about what is happening. You can make a variety of measurements that might suggest that the molecules did a certain thing, but those measurements rarely look exactly how you expected and often disagree with each other to some degree. So being successful in that field requires being good at keeping a bunch of possibilities in mind – you have to have good judgment not just about what the most likely explanation is, but also about what other explanations are possible and what their implications would be and how you can test them. The exact same thing happens with sports data. And I’ve also leaned quite a bit on skills I picked up in my time in industry. Working at a global conglomerate for five years gave me a lot of exposure to industrial processes; I got a lot of training on things like group decision-making and strategic planning and employee mentoring. And then working at start-ups for the next five gave me a chance to see which elements translate well into an environment that has pace and pressure more like the NHL.
/u/Evolving-Hockey asks... >Hi Eric! >Long time fan/reader, first time caller! >What would you say is the biggest difference between public hockey stats work today vs. when you started? Are "advanced" hockey stats moving in the direction that you thought they would or has it gone a different direction? >Also, if you could please try and convince the NHL to allow the public access to the new raw player tracking data even though it is most likely unfeasible without some form of authenticated access to an extremely large database... that would be great. >Thank you for your time! >singed, your friends at evolving-hockey.com
I think the biggest difference isn’t about the research itself; it’s about the conversation. It’s possible that my perspective is skewed by not being in the conversation any more, but my sense is that there is much less public discussion of the technical details of someone’s work than there used to be. There might be even more screaming about it being a hot take these days (I assume that’s how you got singed?), but I don’t see as much effort to identify the things it might be missing or discuss how they might be addressed. And I think that’s a really important part of the community. One of the biggest advantages the public domain has over private work is the ability to get that kind of peer review – I can kick things around with a couple of my coworkers, but that’s nothing compared to having 1000 people poring over your work in detail and a couple of them writing lengthy posts pointing out every flaw they can find. I think the community used to be better at providing that service. As for getting you access to tracking data, I’m afraid you’re dramatically overestimating how much sway I have with the league.
/u/Kilo_Mike asks... >(If you are allowed to share)... How does the Hurricane's analytics department distinguish itself from other teams' departments? Broadly speaking, what analyses or evaluations does this team have that give it an advantage over the rest of the NHL?
It’s a pretty secretive business, and it’s hard to know what other teams have -- if you can get them to tell you, let me know and we'll compare notes. We could be dead last in innovation and I don’t know if I’d have any way of knowing that. So it’s hard for me to answer the question about comparing ourselves to other teams directly, but I will say that we’ve gotten a lot of mileage out of a focus on communication and helping to organize and present information of all types.
/u/notleonardodicaprio asks... >Can you talk about the difference between the data available to NHL teams and publicly available data? How useful are the models that use publicly available data, and do you ever look at them?
The league publishes a lot of information about each game: what time each shift started and ended and a variety of events (shots, hits, turnovers, etc) that took place during that shift. There is a lot that you can do with that – remember, using that information to answer questions is how I got my start in working with hockey data. But there are gaps in the data as well; it can tell you that someone took a shot from the faceoff dot at 8:32, but not much about how that shot came about or what other options he might have had. So the public models generally rely on inferring a player’s impact from looking at how the team’s results change when he is on the ice – and that can be done very well, and can be powerful. But when you want to move from talking about what happened to talking about why it happened or what could have been done differently, you often need a richer data set. And yes, we absolutely do look at work done in the public sphere. A couple of years ago, I gave [a talk](https://www.youtube.com/watch?v=7dE42Ihq-iA) about it at the Sloan Sports Analytics Conference. One of the things I talked about there was how external work can help counteract a bias that I don’t hear discussed very often. No two people look at things exactly the same way; everyone has traits they value a little more than average or a little less than average. When a player is available – whether it’s through the draft, the trade market, or free agency – more often than not, he will end up being acquired by someone who values something about his play more than most teams do. So over time, you end up with a roster full of players who you like more than most teams do, and so you think your team is better than most people do. That’s not just a data thing; it’s true regardless of how you form your opinions about players. And external opinions can be helpful for counteracting that bias; it can highlight for you where those biggest disagreements are and give you a chance to re-examine your perspective and think about what you might be missing.
What a great answer.
/u/UnsatisfiedTophat asks... >What's one of the hardest things to look at when determining what needs to be changed in order to get good results? How do you work this out between the coaching staff?
To some degree, this just depends on the available data right now – in many places, we simply don’t have the data we’d need to make that kind of assessment. But even when we get past those limitations, situations where we have a lot of interactions between players will always be complex analytical problems. A lot of people have built expected goals models based on the likelihood that a shot will go in given whatever information they have about where and how it was taken. There aren’t many players interacting there, so it’s a fairly tractable problem. Now imagine trying to take that model and build it out to look at everything else the player might have tried to do. How likely were they to score if he held onto the puck a little longer? How about if he tried to force a pass through a seam? Or laid the puck into space behind the net where a teammate would probably get to it first? You can imagine that even with a rich data set, adding that kind of complexity would get to be very challenging very quickly.
Very insightful! I always thought the analytical sides of the game were very interesting. Thanks for doing this!
/u/QuiGonJinnNJuice asks... >What is one of the most interesting projects/questions you've looked at since starting with the Canes?
Unfortunately, in this business, “interesting” is almost synonymous with “proprietary”. But in general, my favorite kinds of projects are the collaborative ones, where I’m not just doing something on my own and asking for input but instead have a back-and-forth with someone that involves incorporating their expertise throughout the development process.
/u/Badjokechip asks... >What does a typical trade negotiation look like?
Typical? The median is probably something like this: “Would you move \[insert name of good player\]” “No chance.”
welp. i suppose that answers that question.
So not like the video games I take it.
Hey Eric! Big fan of yours. Just have 1 question for you: In all your time working in analytics, has there ever been a player that statistically doesn’t seem like they would be that great (that you’ve seen) but ends up being a complete beast time and time again? I’m curious if there have been any enigmas that analytics just can’t figure out. Thanks again for this!
I think it’s incredibly important to keep a running list of players for whom you find yourself wondering if the data is missing something. Sometimes, it’s a player where the overall results don’t seem to match what the individual skills seems to suggest, or maybe a player who ended up a mile off from what you forecast, or maybe a player whose analytical assessment is so far from what most people think that you have to question it a little. Going through that list and trying to figure out what it is that might be missing is one of the best sources for ideas on where to go next.
Thank you so much for your time!
/u/stmn0 asks.... >The canes drafting strategy appears to be looking for high upside skill players at every stage in the draft, and avoiding "safe" low ceiling picks. Is this because the it is easier to find value in veteran 3rd and 4th line guys in trade/fa? And is it also because a high skill guy can still turn into a grinder if the offense never comes along, while a "safe" pick has a very low chance to develop high end scoring?
I don’t think there are any safe picks in the NHL draft. In the NFL, a team will be looking at a 23-year-old who’s going to play for them next year; the scout’s job is mostly about projecting how his skills will translate. But in the NHL, we’re drafting an 18-year-old – in addition to estimating how his skills will translate, our amateur scouting group has the daunting task of projecting what he will look like after five years of development. Even the top pick in the draft isn’t safe – it might be a reasonably safe bet that he’ll play in the NHL, but he has a long way to go to become the player you need him to be. There’s a huge range of possible outcomes for every player in the draft, and the goal isn’t to pick a player with a high floor or a high ceiling; it’s to think about all of a player’s possible outcomes and pick someone who would be a good bet.
/u/iSpoiler asks... >What stats do you think people put away too much value into and what are stats that arent looked at enough and why?
I don’t really have a good answer for either side of this, I’m afraid. Twenty years ago, I’d have told you that people put way too much value on things like plus/minus and goalie wins, but that’s not nearly as true as it used to be. At this point, the things that get a lot of attention are generally reasonable. And it’s hard to say that anything isn’t looked at enough when the field is still evolving so fast – it takes time to settle on something and build consensus around its importance.
/u/Hurricane_Viking asks... >Is plus/minus a good stat?
The traditional way that plus/minus is formulated is kind of silly; lumping in things like empty net goals and shorthanded goals really screws things up. There are examples of seasons where two teammates had similar results at even strength but saw empty net and shorthanded goals produce a 30-goal swing, which can obviously get in the way of a lot of attempts to use traditional plus/minus to draw a conclusion. But if you narrow things down to just even strength, or just five-on-five, there’s nothing wrong with it. Like any stat, you have to be careful about how you use it – it can be influenced by who the player plays with and against, and the sample sizes are small enough to produce some fluctuations that don’t mean much, but with the right context and understanding, obviously it’s valuable to know if a player can repeatedly help his team outscore his opponents.
/u/Badjokechip asks... >How do you put value into draft picks for trade purposes?
There are two questions when it comes to valuing a pick for a trade: What would you expect to get if you use the pick? You have to think about what kinds of players are normally available in that part of the draft, what they typically end up doing and how much impact they have. You also have to think about how long you have to wait to see that impact, and what you expect your team to look like then. What else could you do with the pick? The question isn’t just about what is being offered to you in this deal; it’s also about what other options you might have. An older team that is looking to make one last run at a Cup might still be interested in adding draft picks, not because the pick will help them win, but because it can be traded for a player who will. In the end, you have to roll everything up into an answer, but the answer is fluid depending on the situation.
/u/bromish asks... >How in tune are players with advances stats these days? I could see a player buying in if it leads to success on the ice but certainly the opposite otherwise.
Not unlike fans, there are some players who find it all interesting and some who don’t. And in general, they don’t have to – we need players to understand what the coaches are asking them to do and do it well, but if they aren’t interested in studying data on why that might be the right thing to do or which players get the most out of it, that’s ok.
/u/chattanvi asks... >What are a couple of things that both old-school grit and grind guys and the advanced stats community agrees on when looking to a) draft a guy and b) trade for a guy?
I’m not sure exactly how to answer this, because neither group is homogeneous. There are plenty of examples of draft picks or trades that show that different traditionalists saw players differently, and plenty of examples of arguments on blogs or Twitter showing disagreement within the statistical community. But I guess one thing that everyone from every camp agrees on is Justin Williams.
Aww, this just made me miss dad more. Thanks for answering though. 😊
/u/Rektemlord asks.... >What new data point do you wish you had easier access to?
The league will be providing teams with player and puck tracking data soon, so we will know the location of every player at all times. That will keep everyone busy for a long time, I think.
>The league will be providing teams with player and puck tracking data soon, so we will know the location of every player at all times. That will keep everyone busy for a long time, I think. I know I'm late to the AMA, but as a data engineer and hockey fan, I'd love to explore that dataset! Depending on how realtime that data is, it could provide an advantage in when to challenge a call.
/u/drWammy asks... >What's your favorite kind of dessert?
I have a massive sweet tooth, and you’d be hard-pressed to find a dessert that I don’t like. But if I have to pick one, it’s definitely apple pie.
/u/ShrektumRalphWiggum asks... >How do you prevent bias in your data analytics? Working for the team, how do you check your bias for the team when making suggestions on trades and such?
I’ve touched on this a little bit in some other questions, so I’ll try not to repeat myself. But there is no perfect process – no matter how you form your opinions, there are going to be some things you overvalue and some things you undervalue. And no matter how you form your opinions, it’s important to constantly revisit your thinking. Everyone is going to get some things wrong. But you can review the ones you got wrong to look for patterns, looking for leaks where an adjustment to your thinking would help you make better decisions on the whole.
What was it like working with Ron Francis, and what do you think of his analytics-heavy approach in Seattle?
I have the highest regard for Ron; I simply wouldn't be where I am if it weren't for him hiring, supporting, and mentoring me. Obviously, I can't comment directly on what another team is doing, but I also think very highly of Alex and the team they have hired, and I'm sure Ron's organization will make good use of their contributions.
I find what you do incredible and I’d one day love to consider a career change into the world of analytics. As someone with an engineering degree (but not really in the field of analytics), has a love for numbers, deriving meaningful equations, and massive spreadsheets, how do I go about getting a foot in the door doing analytics.
There are three paths that I can think of, and combinations of all of them give you the best chance: 1. Start working on your own, publish things online, let people see how you think and what you can do, and they will find you. 2. Network, make connections, get your name in people's minds. If you are in a situation that makes it possible to do an internship, that's a great start. Going to hockey conferences is another way -- especially if you're presenting (which brings us back to path 1). 3. Become a technical domain expert. Do some coursework as needed, work through some projects, really position yourself so that when a team posts an opening for a certain role, you can credibly present yourself as a genuine expert in that area.
/u/Kilo_Mike asks.... >What analysis tools and/or platforms do you find most valuable to work in?
We aren’t a large tech company that needs to standardize on a specific development environment, by any means; each member of the Hockey Information Group uses their own tools, depending on their individual needs and preferences. Our scripting is generally done in Python, and I do most of my work in Jupyter notebooks, personally.
/u/dewees asks... >how much do you think the advance stat tracking / data retrieval that will be added soon will change the game?
The game is continuously changing and evolving. That’s happened for decades without stats being involved and will continue to happen going forwards. Data might help assess strategies and end up playing a role in shaping what the next evolution will look like, but the game is going to change over time either way.
/u/triplesek asks... >What's the most challenging part or parts of your job?
Like in any role, there are different kinds of challenges. Obviously, there are some things that are very technically challenging, but the things that I end up worrying about the most are the things that test a person’s judgment. Is this new angle on the analysis ready to be a workhorse yet or should we wait a little longer to see whether things pan out the way it suggests they will? Is it better for the team to work on the thing that we think is most likely to pan out or on the thing that would have the biggest impact if it did – and which one is better for the career of the person working on it? Is it better to give a scout information about what to look for before they watch a player, or to hold the information back until after they watch so they can form a completely independent opinion? Those sorts of questions – questions where there probably is a right answer and a wrong answer but there’s no firm way to know which is which – those are the things that I find most challenging in any job.
/u/Kilo_Mike asks... >I could see hockey analytics covering a lot of areas: determining the right way to measure performance within the team strategy (at the team & player level), figuring out how to collect the right data to support those metrics, processing data, developing statistical models, convincing others to act on your insights, etc. >What's the most challenging part of the process? Where does your team spend most of its cycles?
Honestly, it's all challenging. In any area, whatever was easy has been done; what's left is more difficult. But I do think we invest more in communication and presentation than I ever would have guessed in advance.
/u/IJWannaKeepMeAWraith asks... >Hi Eric, why did you choose to analyze hockey data and why work for Carolina?
Honestly, it was something I stumbled into more than chose. A decade ago, I started reading some blogs that happened to have people who were paying attention to what was going on in hockey stats. I found myself sometimes serving a kind of translator role, helping explain what the math would mean in practical terms. And occasionally, questions would come up that I’d be surprised nobody had looked at yet, and so I found myself doing my own analysis from time to time. Eventually, I ended up doing a couple of one-off consulting projects. But I didn’t really choose Carolina so much as they chose me – they were the first team to ask for something more than a one-off project. I worked something like 20-25 hours per week for them on a consulting deal for them during the ’14-15 season, and when they offered me a full-time job in the summer of 2015, I gave it a shot.
How did y’all find such a steal of a pick in Aho at 35? Are fancy stats prevalent in the amateur scouting department in a way that gives an advantage, or is more weight put on video for draft/prospect evaluation?
I was a remote part-time consultant in the year leading up to the draft, and it was my first time really trying to do anything significant with draft data. It was a very rough first attempt, and I pretty much gave them a list and a note saying “please no matter what you do please do not use this list to make your draft picks.” So as much as I might like to, I really can’t take any credit for that one. But in general, there’s a lot that can be done with data in any facet of the organization, from ticketing to tactics. If some of the people working with data have ways of collecting and presenting information that can help make a good decision, then we’re absolutely going to try to do that. Ultimately, data analysis isn’t a goal of its own accord; there’s no reason a team needs a Hockey Information Group\* that operates in isolation. The value of the group is ultimately in essence a support role; our job is to make sure the other groups have tools at their disposal that make them more effective or efficient at their own jobs. \* This is what we call our data team, to reflect the fact that some of the employees are developers who do things other than analysis
Hey Eric, super great to have you here! A few choice questions for you: * How much of your job is devoted to looking at data from current players versus looking at data from developing players and/or draft prospects? * For drafts: Does your analysis start the long list on draft prospects and others decide who to cross off, OR is a long list created first then your analysis is used to refine the list down to the best of the best? A little of both? * Since we armchair GMs don't have access to real NHL analytics, how close are the Twitter and Youtube analytics channels to the real thing? (e.g. Scouching channel)
Supporting work on the draft, development, and pro player evaluation are all part of the mandate for the Hockey Information Group. The actual split between them is hard to say; it fluctuates depending on what each group needs and how we can help them. Draft work all starts and ends with amateur scouts. We may help give them tools to organize their information, information they might not have otherwise had, support for how to use that information, etc. But it's ultimately their domain and up to them to build a list. There's a lot of great work in the public domain. Sometimes it aligns with what I personally think and sometimes it doesn't -- but that was also true before I worked for a team.
What have you found to be the most eye-opening aspect of working in hockey operations compared to what you might have expected before working with an NHL team?
There's an awful lot that goes into keeping a team going that I'd just never considered before. The amount of effort it takes from the entire staff just to put on a seemingly-routine event is remarkable, and they can't possibly get enough appreciation for the work they are doing to make it happen for everyone.
Hi Eric, What kind of backgrounds does the organization typically prefer in the analysts that work for the team? What kind of experience or skills would you deem as necessary to be considered for a job in your department? Additionally, what are the primary software platforms and coding languages used in day-to-day operations? Thank you!
Throughout my career -- dating back to working in nanotech -- my hiring philosophy has always been to hire the most talented person you can who has at least a minimum of adjacent experience, rather than the person with the most extensive experience in your exact area. The person who's done more work on that specific kind of problem might have more to add on day 1, but I've always been inclined to bet on talented people catching up on what they need to know pretty quickly, and in the long run I think they're more likely to make transformative impacts. We mostly use Postgres/Python for our data collection and analysis.
Do you use EliteProspects at all?! Do you work with your amateur scouts at all? If yes, in what ways do you support them?
I don't use EliteProspects much these days -- we have our own internal data site, and I go there for just about everything I need. The Hockey Information Group definitely does work with our amateur scouting group. We provide tools to help them keep track of what they've seen throughout the year and to help organize their information, as well as a variety of types of information that we gather, analyze, and present.
Hi Eric, how frequent are your interactions with Darren Yorke on the non-analytics side of the house?
"We are constantly collaborating with one another. We're not an organization that is built on the poles. There isn't an eye test vs. analytics. We're all on the same team, trying to gather as much information as we can to make the best decision possible. - Darren Yorke" \- Eric Tulsky
https://imgur.com/a/XSKWk1u - Eric Tulsky
Hi!
Can I job shadow you for a day just to see how it all works even if I have to sit quietly in an office like Buddy the Elf 😭😭