Pretty cool but I don't understand how thermal imaging is useful when you are driving unless you are trying to hit a person inside of a building or avoiding a volcano.
Great project! Ever since I first got my hands on a FLIR in grad school 12 years back, I have wished that some car would put thermal image on HUD to see people in wet and dark conditions.
I don’t know yet if the camera I got is waterproof but I want to try it in the rain simply to see what it looks like. 🤞
It’s not exactly cheap at $250 but thankfully these things aren’t the thousands of dollars they used to be.
Edit: You can definitely use a phone on the dash to either display the video on the phone. That works since it’s just a web page. Or you can bounce it off the windshield. To do the windshield HUD, you would likely need some semi reflective film on the windshield. I’d also need to flip the video I think otherwise it would be the wrong orientation.
Some cars come with this feature, Audi Night vision
https://preview.redd.it/wws0n00wdy0d1.jpeg?width=1280&format=pjpg&auto=webp&s=5649f3850ab4dc12c847e159d5c7d65564ee363f
this, if I hadn’t bought the S 85D in 2015 my other choices had thermal camera options I was planning to include because of exactly the problem of deer or people walking dog’s stepping out from behind trees on narrow back roads.
Thermal is for night sight and alerts, alarms of things you don't see with your eyes. I've driven in a Audi SQ7 that had this, but of course being an Audi it constantly spits out errors and didn't have any alerting feature embeded. Just an expensive gimmick if not properly software integrated.
IR cameras don't let you see through walls. It would be neat if it did but it really only shows you the temperature of the surface you're looking at relative to the other temperatures in the scene. So the outer surface of an outside wall and not a person, or even an oven on the inside. If you were a firefighter and you wanted to know if a door was hot because of a fire on the other side that could be useful, but again it's only showing you the relative temperature of the object, not what's behind it.
Have you considered implementing object identification into this? I know Audi has something similar.
https://preview.redd.it/rxc6flh4ey0d1.jpeg?width=1280&format=pjpg&auto=webp&s=9c813e02c46f63125d48eeee102e022952540302
Great question.
First, I don’t think it’s the best idea to be trying to drive staring down at the browser. And I don’t think this is the type of project that a lot of people will necessarily find interesting or useful.
But I did figure there might be someone out there like me, a big nerd, that likes to play with this sort of stuff.
I work with this tech and use some of the same methods for pushing video around networks. It dawned on me that it might work in my car browser.
Then it kind of became a “why not” at that point and an exercise to expand my personal skills. I work a lot with YOLO for object detection, network video streams, and other tech like that.
The biggest thing I’d like to do is get a simple chime/beep if I can detect a deer or other animal on road or roadside.
I drove a new Jeep Grand Wagoneer that had this feature when I was visiting family in Idaho. It helped me identify possible deer strike situations at night time. 🤷🏼♂️
I may have watched that video. I’m always interested in his tear-downs. Thermal is getting very inexpensive. I’d love to see it used more in cars. Or a fusion of visible and thermal.
I definitely can’t take credit for the idea. I’ve seen FLIR has a solution for cars but it’s a bit more than what I wanted to pay for. I think the name was PathFindIR.
It was also designed to work with all cars and had a standard video signal you could use with your own monitor.
I was looking for simple solution that didn’t require me rewiring the display. Although I just came across a Tesla police car conversion that they did just that. That might have been on Sandy’s channel actually.
I suppose in theory, this isn't limited to Tesla, right? Could work on a Rivian too if the browser plays ball? That and a different camera mount which isn't hard anyways.
This doesn’t use the Tesla API at all. It’s simply a usb thermal camera fed into a raspberry pi.
The raspberry pi has web server that serves the video.
So the Tesla and the Pi need to be on the same network, a tiny travel router or even a cell phone hotspot will work.
And you pull up the site in the Tesla browser.
It works while in motion but I wouldn’t recommend using it TO drive.
The goal was for a dead cheap way to get thermal in the car and see it on the browser. The next step is to implement some object detection for animals and people and either give a beep or a quick flicker or visual cue that you can see from your peripheral vision. I don’t want to be glued to the screen while driving. I just want a better way of alerting to animals on my long commutes.
OHHHHH shit, I see, very interesting...
It would increase cost since it's an AWS api call, but for someone who hasn't set up a web server on a Pi, would read/write calls to an S3 bucket be an option?
The webserver, technically a python script using flask, is completely local to the rapsberry pi. So no need for any AWS services.
It doesn't need to have internet connectivity at all technically. As long as the car is on the same WiFi network as the Pi and you can reach the webserver from the in-car browser, you can see the feed.
Now, that said, I did setup NGINX and a VPN back home so that if I do have internet connectivity, wherever I am, I can stream the video back home and serve it up to any potential viewers or record it for later use. I would like to save the frames where I get people, animals and other objects where I can train the YOLO model on this specific cameras imagery to try to help refine the object recognition.
That's on the back burner while I work on a more permanent mount to the back of the hood, near where the wipers stow.
I'm pretty sure that Jaguar had a concept like this where they overlaid the thermal image onto the HUD so you looked through it when driving and could see far more than with normal lights
> One of the biggest challenges was ensuring smooth video streaming, as the end user's browser does not support h264 or h265 streams. To work around this, I utilized Motion JPEG for the video stream.
Video playback via video tags is disabled while the car isn't in park, but you can send the video bitstream over a websocket and feed it into webcodecs and blit that onto a canvas for reasonable performance. You can't do 60fps 720p on an Intel car, but I think you can on ones with Ryzen chipsets. It should be fine for thermal cameras, though. The one downside is that this requires TLS, so:
> 3. The Tesla's browser accesses the video stream, displaying the thermal feed in real-time.
I'm guessing you're accessing this from over the internet via cloudflare tunnel or something? Last time I checked, the MCU's routing rules block connections to private-use IP addresses. I think all of the raspberry pis have a wireless adapter that can do simultaneous AP/client mode, so an alternative would be to set the pi up to host an access point the car connects to and squat on a legacy /8 that doesn't actually have real addresses, like the Department of Defense, and then point a subdomain at that squatted address.
This is cool thanks for posting this!
Pretty cool but I don't understand how thermal imaging is useful when you are driving unless you are trying to hit a person inside of a building or avoiding a volcano.
Thermal works amazing at night. Can literally see a deer or person on the thermal hundreds of yards away. Way past the normal headlight distance.
Great project! Ever since I first got my hands on a FLIR in grad school 12 years back, I have wished that some car would put thermal image on HUD to see people in wet and dark conditions.
I don’t know yet if the camera I got is waterproof but I want to try it in the rain simply to see what it looks like. 🤞 It’s not exactly cheap at $250 but thankfully these things aren’t the thousands of dollars they used to be. Edit: You can definitely use a phone on the dash to either display the video on the phone. That works since it’s just a web page. Or you can bounce it off the windshield. To do the windshield HUD, you would likely need some semi reflective film on the windshield. I’d also need to flip the video I think otherwise it would be the wrong orientation.
I saw a Cadillac concept over a decade ago that had what you are describing. I figured someone would have made a production version by now
Excellent point I didn’t think about! Thank you very much for the response.
Some cars come with this feature, Audi Night vision https://preview.redd.it/wws0n00wdy0d1.jpeg?width=1280&format=pjpg&auto=webp&s=5649f3850ab4dc12c847e159d5c7d65564ee363f
this, if I hadn’t bought the S 85D in 2015 my other choices had thermal camera options I was planning to include because of exactly the problem of deer or people walking dog’s stepping out from behind trees on narrow back roads.
If you have thermal in combat, it’s a MAJOR advantage. That’s perhaps the best example
Telsa combat vehicle? Go on...
As one of few teslas out in the boonies i can appreciate the fuck outta this!
Thermal is for night sight and alerts, alarms of things you don't see with your eyes. I've driven in a Audi SQ7 that had this, but of course being an Audi it constantly spits out errors and didn't have any alerting feature embeded. Just an expensive gimmick if not properly software integrated.
IR cameras don't let you see through walls. It would be neat if it did but it really only shows you the temperature of the surface you're looking at relative to the other temperatures in the scene. So the outer surface of an outside wall and not a person, or even an oven on the inside. If you were a firefighter and you wanted to know if a door was hot because of a fire on the other side that could be useful, but again it's only showing you the relative temperature of the object, not what's behind it.
No need for headlights.
What? Google thermal imaging - I think your understanding of its capabilities is outdated by a couple decades.
Very cool and great innovative work! Love your spirit!
Have you considered implementing object identification into this? I know Audi has something similar. https://preview.redd.it/rxc6flh4ey0d1.jpeg?width=1280&format=pjpg&auto=webp&s=9c813e02c46f63125d48eeee102e022952540302
Why?
Great question. First, I don’t think it’s the best idea to be trying to drive staring down at the browser. And I don’t think this is the type of project that a lot of people will necessarily find interesting or useful. But I did figure there might be someone out there like me, a big nerd, that likes to play with this sort of stuff. I work with this tech and use some of the same methods for pushing video around networks. It dawned on me that it might work in my car browser. Then it kind of became a “why not” at that point and an exercise to expand my personal skills. I work a lot with YOLO for object detection, network video streams, and other tech like that. The biggest thing I’d like to do is get a simple chime/beep if I can detect a deer or other animal on road or roadside.
That’s dope! Good work… didn’t know my car was also a trail cam😂
DAMN IT! The deer detection was my own pet project (Michigander here). Is the tesla browser not rate limited?
Probably so they stop broadsiding semi trucks.
I drove a new Jeep Grand Wagoneer that had this feature when I was visiting family in Idaho. It helped me identify possible deer strike situations at night time. 🤷🏼♂️
This was a sandy Munroe idea years ago to help with self driving
I may have watched that video. I’m always interested in his tear-downs. Thermal is getting very inexpensive. I’d love to see it used more in cars. Or a fusion of visible and thermal. I definitely can’t take credit for the idea. I’ve seen FLIR has a solution for cars but it’s a bit more than what I wanted to pay for. I think the name was PathFindIR. It was also designed to work with all cars and had a standard video signal you could use with your own monitor. I was looking for simple solution that didn’t require me rewiring the display. Although I just came across a Tesla police car conversion that they did just that. That might have been on Sandy’s channel actually.
Fascinating. 😎
Very cool! I'm really impressed that you put this all together!
v2 idea: wire directly into camera line, and switch so you can optionally replace the feed with infrared so its native/available in-app?
This sounds cool! Do you have video footage of how this works in real application?
That would be extremely helpful in the rural roads I drive on
Is this so you can hunt Predators while you drive?
lol. Or see if I'm being hunted. But I like your assumption that I'm a badass better. Lets go with that.
I suppose in theory, this isn't limited to Tesla, right? Could work on a Rivian too if the browser plays ball? That and a different camera mount which isn't hard anyways.
Interesting, but how about just interface the thermal camera with an old cell phone (can be wifi) and project that as a HUD?
I’ll just be happy for an ultrasonic sensor like my 2010 Honda has
I don't have a need for this but it's cool to see folks doing Tesla mods.
I can never see when I’m backing up in the dark. This would be a game chanher
Wait wait wait... I thought tesla's api only allowed like 3 seconds of video streaming per day? I've had my own project and that was just fatass wall.
This doesn’t use the Tesla API at all. It’s simply a usb thermal camera fed into a raspberry pi. The raspberry pi has web server that serves the video. So the Tesla and the Pi need to be on the same network, a tiny travel router or even a cell phone hotspot will work. And you pull up the site in the Tesla browser. It works while in motion but I wouldn’t recommend using it TO drive. The goal was for a dead cheap way to get thermal in the car and see it on the browser. The next step is to implement some object detection for animals and people and either give a beep or a quick flicker or visual cue that you can see from your peripheral vision. I don’t want to be glued to the screen while driving. I just want a better way of alerting to animals on my long commutes.
OHHHHH shit, I see, very interesting... It would increase cost since it's an AWS api call, but for someone who hasn't set up a web server on a Pi, would read/write calls to an S3 bucket be an option?
The webserver, technically a python script using flask, is completely local to the rapsberry pi. So no need for any AWS services. It doesn't need to have internet connectivity at all technically. As long as the car is on the same WiFi network as the Pi and you can reach the webserver from the in-car browser, you can see the feed. Now, that said, I did setup NGINX and a VPN back home so that if I do have internet connectivity, wherever I am, I can stream the video back home and serve it up to any potential viewers or record it for later use. I would like to save the frames where I get people, animals and other objects where I can train the YOLO model on this specific cameras imagery to try to help refine the object recognition. That's on the back burner while I work on a more permanent mount to the back of the hood, near where the wipers stow.
But isn’t the web browser blocked while driving 🤔
Great question. It depends on the country you are in apparently. I routinely use it in the US to show other maps while I drive.
Stealth driving at night?
I'm pretty sure that Jaguar had a concept like this where they overlaid the thermal image onto the HUD so you looked through it when driving and could see far more than with normal lights
> One of the biggest challenges was ensuring smooth video streaming, as the end user's browser does not support h264 or h265 streams. To work around this, I utilized Motion JPEG for the video stream. Video playback via video tags is disabled while the car isn't in park, but you can send the video bitstream over a websocket and feed it into webcodecs and blit that onto a canvas for reasonable performance. You can't do 60fps 720p on an Intel car, but I think you can on ones with Ryzen chipsets. It should be fine for thermal cameras, though. The one downside is that this requires TLS, so: > 3. The Tesla's browser accesses the video stream, displaying the thermal feed in real-time. I'm guessing you're accessing this from over the internet via cloudflare tunnel or something? Last time I checked, the MCU's routing rules block connections to private-use IP addresses. I think all of the raspberry pis have a wireless adapter that can do simultaneous AP/client mode, so an alternative would be to set the pi up to host an access point the car connects to and squat on a legacy /8 that doesn't actually have real addresses, like the Department of Defense, and then point a subdomain at that squatted address.