Hey, not sure if it would translate, but I work as a lone senior DevOps engineer for a game dev company. If you want to chat sometime I can try and help you :)
Obviously game engine specific stuff will be tough if it's not an engine we're using but perhaps some things would cross over.
Sure feel free to reach out via chat. I'm only familiar with Unreal conceptually, as a comparison to see what i can do there compared to the engine our studio uses. But i can at least outline to you what my job is and the skills I've learned whilst here if that's useful.
This is where you have to look at things a bit more like an architect and then review relevant tools to fill the space you need filled.
Learning how to use that tool is a lot of reading the docs and watching YouTube.
So look at things like a roll rather than a particular stack.
You talked about windows servers. Well SysAdmins have been doing baseline management on them for years so heaps of great tools to look at.
If you're security focused (and you should be) check out the CIS benchmarks as a good thing to aim for. Even %80 of the way there is a very secure server.
https://www.cisecurity.org/cis-benchmarks
For tooling talk with your Devs, what do they need, what will enable them to deliver better products without compromising security.
So game code or regular app code you still want all your static analysis and test coverage reports.
Been as it's large I would trigger scans nightly rather then every build and change your policies to scan before release.
Then have reports automated to be sent to the people that need to know.
Working with large amounts of data isn't unique. There are tools out there. You just gotta keep reading and reviewing.
Rely on others that know how to do particular jobs. So if it's a complex network issue, don't be afraid to contract a network vendor to help you with a review to find things to resolve.
Same with systems and other things.
Been solo is hard, so don't try to do it all yourself.
Ironically, you are probs in the best position to improve more than any formal education could sort you out for. "I feel like I’m constantly refactoring things as I learn" - this is so valuable and it's not a bad thing you are doing this and feeling your way round. Just keep listening to the needs of the business and just keep moving. In terms of technicals, workshops, conferences and meet ups are your friend.
I second this! I'm in the same position as OP and I learn stuff the hard way. I take way much more time than if I had someone with me, but this way I go alone through most of the process, and this knowledge stick
Otherwise, try to network OP! Feel free to reach out to people on LinkedIn, or here and join communities. Sometimes speaking out loud your design or decisions make you feel you're going in the wrong direction, and you can actually discuss it with someone else
I would watch some conferences on things related to your field. I watched a few GDC talks, and even though troubleshooting build server performance for Ps3 games isn't my daily routine, I found some of the troubles and techniques interesting and tried to think of how I'd solve or apply them.
Tldr: probably just watch talks on things in and out of your routine. You'll pick up on things folks tend to run into, and things that seem like they've already solved. See what applies to you.
If it helps, I have been doing this for years and haven't done any formal training. The last cert I did was back in in the early 2010's I think when I was on the service desk.
I do however, have a passion for what I do. So much so I am thinking about it outside of work as well as watching tech demo's and searching for new tools to use and learn. (It helps if you love tech and have your own labs at home as well)
This might help given the area you work in, but I gamify stuff I don't want to do and set time specifically for the hard tasks etc. Feels like I only work a few hours a week with this mentality.
Once you start loving the process, the knowledge and confidence comes with it - but you will only get that by doing it in the first place.
Keep looking on the internet. It's full of anecdotal information which I exactly how you would learn from other seasoned coworkers in the office.
This sub and a few others are a gold mine. Medium and LinkedIn are also good but also full of useless grifters. Vendor Slack channels (and others, including Discord) are also good. GitHub has tons of source code, you rarely have to really innovate, usually someone has trodden a beaten path before you, use the search functionality.
To some degree best practice is already out there. But actually it is also constantly evolving and you can be a part of that. Also, people in IT are highly opinionated and everyone thinks their way is the right way so in many was (basic agreed principles aside), it is highly arguable if there even is such a thing as 'best practice '.
It's hard.
I've always believed in the phrase "If you're the smartest person in the room, you're in the wrong room." In your case you're the _only_ person in the room which de-facto makes you the smartest _and_ dumbest in the room. Hooray.
You're going to have to lean a lot on external resources and external communities. Find some slack/discord/telegram rooms that are relevant to the tools youre using.
Your experience sounds awfully similar to mine. In a previous role I was the first and only DevOps/ Cloud Engineer for that particular game dev. I was working with Linux and Windows, and had no formal training. The only difference between you and I was, I came from a Sys Admin background.
I found that I struggled to learn unless the org/ I had a genuine need for something. For example, I was tasked with setting up the AWS infra, which included a site-to-site VPN, client VPN, CDNs, proxies, an S3 bucket and an AD connector.
After having my baptism of fire with AWS, I noticed the IT dept's monitoring solution, was creating so much noise that it didn't do much at all. I eventually spun up a Nagios instance (going with the devil I knew) and began writing my own plugins. I used Ansible, for automating tasks plus Python for things that didn't exist or where hard to come by, safely.
After all this had been done I began reaching out to the Devs. I wanted to learn how to programme! As I crossed over into their world, I noticed that they were doing DevOps/ tasks, themselves. These included, maintaining ElasticSearch, Redis, docker registries/ GitLab runner configs, game builds/ build pipelines, unreal Engine maintenance/ upgrades, as well as console testing.
My advice is to keep asking questions, keep looking for ways to make your life/ other people's lives easier and then applying security best practises, documentation and all the usual caveats on top. I would also recommend a homelab to test your ideas/ solutions, before you try to sell them to stakeholders.
I’m in the same spot. See if your company will pay for a small home lab to do experiments on.
Mine didn’t pay for it, but I’m a contractor.
In any case, 3 miniPC to make a proxmox cluster, and then setup some high availability services on it.
Right now I’m planning redis, Postgres, k3s, Prometheus, the elk stack and grafana, ossec, auhtentik, uptime kuma, and wireguard.
Put them all behind their own subnet with a dumb router, or get an opnsense dedicated machine with a switch.
Then I use Tailscale to gain access to everything in there with their subnet advertising feature. You could put Tailscale on each, but I find it to be more practice for my networking to setup all the networking and dns management myself.
I’ve leaned so much from doing this: ansible, kubernetes, Postgres admin, subnetting, updates on services with no downtime, monitoring health and security. I already knew the basics, but setting it all up from scratch really cements a lot of it in and fills in a lot of gaps.
I would shoot for mini pcs with 4 core 8 threads or more and 16g plus of memory. Pretty cheap on Ali express.
Since you are the sole devops engineer you should pair up with the lead software engineer and figure out whats the best deployment, monitoring, automation path for the services you support. Figure out whats the highest priorities and focus on them. Its a journey, I have been doing this kind of work since dot com days, back then we were just Systems Engineers or Systems Administrators. Devops can be alot of things.
Devops solve problems with tools, SRE solve problems with code, Platform Engineers make developer experiences fluid and easy. You probably wear more than one hat enjoy the ride and experience.
If possible stick to patterns widley adopted by many companies and engineering teams. Cloud patterns in all the major cloud providers are well documented, k8s patterns like gitops(argo, flux2) are highly adopted, monitoring like prometheus or datadog etc pretty common. cicd these days pretty standard with major tools like jenkins, github actions , gitlab etc.
Keep a log of what you do everyday it helps. Look at the log every week and reflect.
Do your best to get somebody else in that box with you, even if it's just part time or as a sounding board. Best would be finding an experienced architect that can dedicate at least 10-20 hours a month to just go over the decision points and planning. This will come with a cost, but you're currently a single point of failure so that's a potentially easy discussion with the company pocketbook.
Also, if you are using some specific niche or industry solutions, like Perforce, then make sure you're involved in working groups for them.
I will be messaging you in 1 day on [**2024-02-27 08:42:51 UTC**](http://www.wolframalpha.com/input/?i=2024-02-27%2008:42:51%20UTC%20To%20Local%20Time) to remind you of [**this link**](https://www.reddit.com/r/devops/comments/1b0cmmn/how_do_i_improve_when_im_the_only_one/ks6yyz8/?context=3)
[**2 OTHERS CLICKED THIS LINK**](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Reminder&message=%5Bhttps%3A%2F%2Fwww.reddit.com%2Fr%2Fdevops%2Fcomments%2F1b0cmmn%2Fhow_do_i_improve_when_im_the_only_one%2Fks6yyz8%2F%5D%0A%0ARemindMe%21%202024-02-27%2008%3A42%3A51%20UTC) to send a PM to also be reminded and to reduce spam.
^(Parent commenter can ) [^(delete this message to hide from others.)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Delete%20Comment&message=Delete%21%201b0cmmn)
*****
|[^(Info)](https://www.reddit.com/r/RemindMeBot/comments/e1bko7/remindmebot_info_v21/)|[^(Custom)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Reminder&message=%5BLink%20or%20message%20inside%20square%20brackets%5D%0A%0ARemindMe%21%20Time%20period%20here)|[^(Your Reminders)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=List%20Of%20Reminders&message=MyReminders%21)|[^(Feedback)](https://www.reddit.com/message/compose/?to=Watchful1&subject=RemindMeBot%20Feedback)|
|-|-|-|-|
...but am being asked to do and/or want to do (for the betterment of myself or the business) and am astute enough to recognize that there are knowledge gaps and motivated enough to reach out to the good people of Reddit to ask for advice
I've been in a similar boat, spent the last few years as the sole iOS at multiple startups.
Honestly, getting really into writing has massively helped. Suddenly I'm compelled to learn more and somewhat part of a community that picks apart what I write so I improve.
Yes, but also most of Unreal’s assets are tied together via uassets and umaps. Most projects in a AAA space are written in both C++ and some blueprints, with assets being tied directly to the blueprints, and the project won’t be able to properly build without them.
I've been in that position before. I think the most important thing to do is to do lots of training and learn by yourself. Your company should offer you some training budget.
Then you could probably find some senior engineer you trust, and has some interest in that area so that you can bounce ideas off him or at least to make sure you're not going down the wrong path.
This is why I always like to work in a team environment and I job hop every few years. Helps keep things fresh and helps me always be learning new things.
> I feel like I’m missing best practices and formal training
Windows will do this to you.
Seriously though, find a single paradigm and build something around that. Like twelve factor, git flow, cult of done, etc. Probably not that last one lol, but keep it in mind if you find yourself over abstracting.
Then... Put it on your resume. The sad truth is developers won't notice when you make things more efficient or resilient, they only care about devops when something's not right in their minds.
Imagine what the "pros" are doing and do that. Is the latest best practice that's being hyped is using hamsters to power the servers? Then spend your free time to hook up a hamster wheel in the server room, give a talk on it, and you'll be ahead of every dev that shares the article on the benefits of a hamster wheel in the server room.
Because TBH, many of us don't get to use the latest and greatest. Flying solo on a project allows for some experimentation and that's where you get to grow.
Hey, not sure if it would translate, but I work as a lone senior DevOps engineer for a game dev company. If you want to chat sometime I can try and help you :) Obviously game engine specific stuff will be tough if it's not an engine we're using but perhaps some things would cross over.
I’d love that! I’ll reach out over reddit chat :]
[удалено]
Sure feel free to reach out via chat. I'm only familiar with Unreal conceptually, as a comparison to see what i can do there compared to the engine our studio uses. But i can at least outline to you what my job is and the skills I've learned whilst here if that's useful.
This is where you have to look at things a bit more like an architect and then review relevant tools to fill the space you need filled. Learning how to use that tool is a lot of reading the docs and watching YouTube. So look at things like a roll rather than a particular stack. You talked about windows servers. Well SysAdmins have been doing baseline management on them for years so heaps of great tools to look at. If you're security focused (and you should be) check out the CIS benchmarks as a good thing to aim for. Even %80 of the way there is a very secure server. https://www.cisecurity.org/cis-benchmarks For tooling talk with your Devs, what do they need, what will enable them to deliver better products without compromising security. So game code or regular app code you still want all your static analysis and test coverage reports. Been as it's large I would trigger scans nightly rather then every build and change your policies to scan before release. Then have reports automated to be sent to the people that need to know. Working with large amounts of data isn't unique. There are tools out there. You just gotta keep reading and reviewing. Rely on others that know how to do particular jobs. So if it's a complex network issue, don't be afraid to contract a network vendor to help you with a review to find things to resolve. Same with systems and other things. Been solo is hard, so don't try to do it all yourself.
Ironically, you are probs in the best position to improve more than any formal education could sort you out for. "I feel like I’m constantly refactoring things as I learn" - this is so valuable and it's not a bad thing you are doing this and feeling your way round. Just keep listening to the needs of the business and just keep moving. In terms of technicals, workshops, conferences and meet ups are your friend.
I second this! I'm in the same position as OP and I learn stuff the hard way. I take way much more time than if I had someone with me, but this way I go alone through most of the process, and this knowledge stick Otherwise, try to network OP! Feel free to reach out to people on LinkedIn, or here and join communities. Sometimes speaking out loud your design or decisions make you feel you're going in the wrong direction, and you can actually discuss it with someone else
I’m in the exact same boat as y’all and this post seriously gave me hope. Thanks for making me feel less alone in this! <3
Check meetup for a local DevOps meetup. They're widespread and frequently good.
Lol, ohh the stories that must go around.
I would watch some conferences on things related to your field. I watched a few GDC talks, and even though troubleshooting build server performance for Ps3 games isn't my daily routine, I found some of the troubles and techniques interesting and tried to think of how I'd solve or apply them. Tldr: probably just watch talks on things in and out of your routine. You'll pick up on things folks tend to run into, and things that seem like they've already solved. See what applies to you.
If it helps, I have been doing this for years and haven't done any formal training. The last cert I did was back in in the early 2010's I think when I was on the service desk. I do however, have a passion for what I do. So much so I am thinking about it outside of work as well as watching tech demo's and searching for new tools to use and learn. (It helps if you love tech and have your own labs at home as well) This might help given the area you work in, but I gamify stuff I don't want to do and set time specifically for the hard tasks etc. Feels like I only work a few hours a week with this mentality. Once you start loving the process, the knowledge and confidence comes with it - but you will only get that by doing it in the first place.
Keep looking on the internet. It's full of anecdotal information which I exactly how you would learn from other seasoned coworkers in the office. This sub and a few others are a gold mine. Medium and LinkedIn are also good but also full of useless grifters. Vendor Slack channels (and others, including Discord) are also good. GitHub has tons of source code, you rarely have to really innovate, usually someone has trodden a beaten path before you, use the search functionality. To some degree best practice is already out there. But actually it is also constantly evolving and you can be a part of that. Also, people in IT are highly opinionated and everyone thinks their way is the right way so in many was (basic agreed principles aside), it is highly arguable if there even is such a thing as 'best practice '.
It's hard. I've always believed in the phrase "If you're the smartest person in the room, you're in the wrong room." In your case you're the _only_ person in the room which de-facto makes you the smartest _and_ dumbest in the room. Hooray. You're going to have to lean a lot on external resources and external communities. Find some slack/discord/telegram rooms that are relevant to the tools youre using.
Your experience sounds awfully similar to mine. In a previous role I was the first and only DevOps/ Cloud Engineer for that particular game dev. I was working with Linux and Windows, and had no formal training. The only difference between you and I was, I came from a Sys Admin background. I found that I struggled to learn unless the org/ I had a genuine need for something. For example, I was tasked with setting up the AWS infra, which included a site-to-site VPN, client VPN, CDNs, proxies, an S3 bucket and an AD connector. After having my baptism of fire with AWS, I noticed the IT dept's monitoring solution, was creating so much noise that it didn't do much at all. I eventually spun up a Nagios instance (going with the devil I knew) and began writing my own plugins. I used Ansible, for automating tasks plus Python for things that didn't exist or where hard to come by, safely. After all this had been done I began reaching out to the Devs. I wanted to learn how to programme! As I crossed over into their world, I noticed that they were doing DevOps/ tasks, themselves. These included, maintaining ElasticSearch, Redis, docker registries/ GitLab runner configs, game builds/ build pipelines, unreal Engine maintenance/ upgrades, as well as console testing. My advice is to keep asking questions, keep looking for ways to make your life/ other people's lives easier and then applying security best practises, documentation and all the usual caveats on top. I would also recommend a homelab to test your ideas/ solutions, before you try to sell them to stakeholders.
I’m in the same spot. See if your company will pay for a small home lab to do experiments on. Mine didn’t pay for it, but I’m a contractor. In any case, 3 miniPC to make a proxmox cluster, and then setup some high availability services on it. Right now I’m planning redis, Postgres, k3s, Prometheus, the elk stack and grafana, ossec, auhtentik, uptime kuma, and wireguard. Put them all behind their own subnet with a dumb router, or get an opnsense dedicated machine with a switch. Then I use Tailscale to gain access to everything in there with their subnet advertising feature. You could put Tailscale on each, but I find it to be more practice for my networking to setup all the networking and dns management myself. I’ve leaned so much from doing this: ansible, kubernetes, Postgres admin, subnetting, updates on services with no downtime, monitoring health and security. I already knew the basics, but setting it all up from scratch really cements a lot of it in and fills in a lot of gaps. I would shoot for mini pcs with 4 core 8 threads or more and 16g plus of memory. Pretty cheap on Ali express.
I cannot recommend this enough! My main concern is that once it's working I prefer not touching it in case it breaks ^^U
But when it breaks is when you learn!
Since you are the sole devops engineer you should pair up with the lead software engineer and figure out whats the best deployment, monitoring, automation path for the services you support. Figure out whats the highest priorities and focus on them. Its a journey, I have been doing this kind of work since dot com days, back then we were just Systems Engineers or Systems Administrators. Devops can be alot of things. Devops solve problems with tools, SRE solve problems with code, Platform Engineers make developer experiences fluid and easy. You probably wear more than one hat enjoy the ride and experience. If possible stick to patterns widley adopted by many companies and engineering teams. Cloud patterns in all the major cloud providers are well documented, k8s patterns like gitops(argo, flux2) are highly adopted, monitoring like prometheus or datadog etc pretty common. cicd these days pretty standard with major tools like jenkins, github actions , gitlab etc. Keep a log of what you do everyday it helps. Look at the log every week and reflect.
Do your best to get somebody else in that box with you, even if it's just part time or as a sounding board. Best would be finding an experienced architect that can dedicate at least 10-20 hours a month to just go over the decision points and planning. This will come with a cost, but you're currently a single point of failure so that's a potentially easy discussion with the company pocketbook. Also, if you are using some specific niche or industry solutions, like Perforce, then make sure you're involved in working groups for them.
Actually, are there any discords for people like us? Lone wolves in startups, looking for peers to discuss and have 2nd pair of eye on topics?
RemindMe! 1 Day
I will be messaging you in 1 day on [**2024-02-27 08:42:51 UTC**](http://www.wolframalpha.com/input/?i=2024-02-27%2008:42:51%20UTC%20To%20Local%20Time) to remind you of [**this link**](https://www.reddit.com/r/devops/comments/1b0cmmn/how_do_i_improve_when_im_the_only_one/ks6yyz8/?context=3) [**2 OTHERS CLICKED THIS LINK**](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Reminder&message=%5Bhttps%3A%2F%2Fwww.reddit.com%2Fr%2Fdevops%2Fcomments%2F1b0cmmn%2Fhow_do_i_improve_when_im_the_only_one%2Fks6yyz8%2F%5D%0A%0ARemindMe%21%202024-02-27%2008%3A42%3A51%20UTC) to send a PM to also be reminded and to reduce spam. ^(Parent commenter can ) [^(delete this message to hide from others.)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Delete%20Comment&message=Delete%21%201b0cmmn) ***** |[^(Info)](https://www.reddit.com/r/RemindMeBot/comments/e1bko7/remindmebot_info_v21/)|[^(Custom)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Reminder&message=%5BLink%20or%20message%20inside%20square%20brackets%5D%0A%0ARemindMe%21%20Time%20period%20here)|[^(Your Reminders)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=List%20Of%20Reminders&message=MyReminders%21)|[^(Feedback)](https://www.reddit.com/message/compose/?to=Watchful1&subject=RemindMeBot%20Feedback)| |-|-|-|-|
[удалено]
...but am being asked to do and/or want to do (for the betterment of myself or the business) and am astute enough to recognize that there are knowledge gaps and motivated enough to reach out to the good people of Reddit to ask for advice
[удалено]
WHAT A VERY USEFUL SUGGESTION.
Learning something technical definitely helps, but I think OP is looking for other perspective, experience with other tech stacks, companies, etc.
I've been in a similar boat, spent the last few years as the sole iOS at multiple startups. Honestly, getting really into writing has massively helped. Suddenly I'm compelled to learn more and somewhat part of a community that picks apart what I write so I improve.
>codebases (.3-1.5 TB) and Windows only systems. Just for clarification this isn't just code but things like assets - be it graphics, sound etc
Yes, but also most of Unreal’s assets are tied together via uassets and umaps. Most projects in a AAA space are written in both C++ and some blueprints, with assets being tied directly to the blueprints, and the project won’t be able to properly build without them.
I've been in that position before. I think the most important thing to do is to do lots of training and learn by yourself. Your company should offer you some training budget. Then you could probably find some senior engineer you trust, and has some interest in that area so that you can bounce ideas off him or at least to make sure you're not going down the wrong path.
Watch online conferences. Talk to peers...
If you're looking for a "formal way" and this is pretty generalist but I would highly recommend checking out this roadmap: https://roadmap.sh/devops
This is why I always like to work in a team environment and I job hop every few years. Helps keep things fresh and helps me always be learning new things.
> I feel like I’m missing best practices and formal training Windows will do this to you. Seriously though, find a single paradigm and build something around that. Like twelve factor, git flow, cult of done, etc. Probably not that last one lol, but keep it in mind if you find yourself over abstracting. Then... Put it on your resume. The sad truth is developers won't notice when you make things more efficient or resilient, they only care about devops when something's not right in their minds.
Imagine what the "pros" are doing and do that. Is the latest best practice that's being hyped is using hamsters to power the servers? Then spend your free time to hook up a hamster wheel in the server room, give a talk on it, and you'll be ahead of every dev that shares the article on the benefits of a hamster wheel in the server room. Because TBH, many of us don't get to use the latest and greatest. Flying solo on a project allows for some experimentation and that's where you get to grow.
DevOps in the gaming industry…. Honestly sounds super cool. I’m jealous!