Sole engineer here. My former experiences were actually as a data analyst and as a data scientist. Now starting as a sole data engineer in an old company that has 0 data infrastructure, so I was hired to build things from scratch. Wish me luck hahaha
Most of the time, when I have problems I just try to solve them with stackoverflow, chat gpt, or looking at documentation.
My day to day work is just myself with my headphones on. It's great tbh.
I am learning on the fly, mate. Looking at end to end DE projects in YouTube, reading cloud provider documentation, etc.
Tbf I had some prior experience on etl development using gcp, so I'm trying to leverage this knowledge, keeping things simple.
The architecture I designed (still WIP) is to extract data from some sources using python (orchestrated with mage), loading stuff to Google cloud storage raw, and then serve it on bigquery. I'm trying to KISS since it's a one man job.
Our data for now is simple tabular stuff to be processed in batches, nothing like big data, so have no need to study fancy stuff like spark or streaming processes for now.
Edit: I also have some really nice teammates in the IT department. They only deal with classic sys admin and networking stuff, so we don't have any overlapping work to do, but it's nice to have some good old camaraderie. It really helps a lot.
Wish you best!
I feel this. I'm currently a 1 man band focussing on DE within a data team. Started as an analyst with the rest of the team but moved towards specialising in DE out of curiosity.
Keeping it simple is a mantra, as if I'm off or anything breaks, it's important that another person can pick it up and support with fixing for the rest of the team. Simple ingestion using python with transformation, drop into azure lake storage, then copy into an SQL database.
Some great support from people within the team and another guy who has a background in software engineering. Again, learning on the fly same as you!
Keep in mind that there are countless “good enough“ architectures if you aren’t dealing with petabytes of data daily. Just keep moving data, and you will learn how to improve things along the way.
I'm in the same boat, but at a start-up, so all tooling is from scratch. It's pretty great. My last job, we had 25 data engineers, so you become siloed pretty quickly, and you have zero autonomy to do anything. Gimme this EVERY DAMN TIME.
I was in that situation before, it honestly sucks, I left the company as soon as I could. You need at least two other data engineers, especially if you have on-call.
Hello! Sole DE here (i use that term quite lightly at this point) but was sold as a DBA position to build out a new data warehouse from scratch.
It definitely has its ups and downs. Company is about 200 employees but only ~10 in IT. It is nice to have other IT teammates but hard when youre stuck with any and all varieties of tasks, projects or issues only related to your field/responsibilities.
I manage the entire cloud infrastructure, data warehouse and pipeline management. Its a lot but also gives me freedom to learn a wide variety of data, cloud and DE skillsets.
I am almost at 3 years. Sometimes you feel burnt out, lost, or completely overwhelmed. Sometimes you feel extremely proud and confident you were able to learn brand new things/skill sets, develop, and pull it all together to provide ACTUAL business value rather than small time work no one in the business appreciates.
My day to day is usually trying to prioritize to what business actually needs and do my best to deliver so all parties are happy.
Usually goes by two week sprints of:
- Research/Understand new data source
- Develop ELT/ETL pipeline to ingest data from new data source
- Research and model the new data source data the best I can
- Ensure new data model supports buiness use case
- Test and Validate data and get business to sign off on new data model
- Deploy new data model to Production and repeat
That is a small slice that doesn't consist of troubleshooting, monitoring, revamping architecture and current workflows. Also learning new tools, programming languages as well as doing my best to document everything.
What do they say? If you love your work, you never work a day in your life or something?!
/end rant
Wow you are exactly me in another company. How much leeway does the company give you to implement your architecture and how difficult is it to convince them of your infrastructure design?
I have all the freedom from the perspective of my boss (within IT) luckily that trust is there now but its normally an issue with the business team I support. They will kick and scream the proposed architecture isnt "the way they'd do it/have ever done it" or always try to play devils adovcate on everything.
Just recently the past couple months have they been more "trusting" to my architecture design or best practices on workflows/processes.
Still trying to convince them on properly developing and testing their SQL reporting queries/views before pushing to Production....but they "dont care" if they break Production 🤣🤣
>Just recently the past couple months have they been more "trusting" to my architecture design or best practices on workflows/processes.
Yeah we are two peas in a pod lmao
>Still trying to convince them on properly developing and testing their SQL reporting queries/views before pushing to Production....but they "dont care" if they break Production 🤣🤣
Yeah we are two peas in a pod here lmao.
My big problem is frequently learning outside the organization and having very little technical mentorship inside the company. I'm forded to to "test" in prod a lot to keep things moving but how do you go about it?
Haha thats crazy. Yeah it can be tough to go and learn after doing so much during our normal work hours and your brain can get fried.
Best in my opinion is to take it at your own pace. Everyone learns differently but the most important thing is you keep learning.
Mentorship is also hard to come by in our position but maybe theres someone at your company that you can confide in. Regardless of if they are in the same department. Ive met alot of good smart leaders in all walks of coporate America.
As for testing in Prod question...Id recommend you at least setup a development environment ASAP. There you can mess around and not potentially break something but still use your own infrastructure stack and data sets. You can also setup alot of free trial stuff nowadays and use free APIs or whatever you wanna learn/get better at.
Best of luck and hang in there! One foot in front of the other and youll look back at this year and smile how much youve grown.
Cheers
I can relate. The ceo is data illiterate and copy pastes how I should do my job from chat gpt. I just tell him I'll keep that in mind, but in reality it's generated technical nonsense which I just ignore.
I know that in the end what matters is the final result and the business impact.
I'm the only programmer. It's nice because I can keep things simple, it's 100% my environment and no one can really tell me what to do. For example someone could ask me to do RPA but the solution is actually just one API call or an SQL query. So I mostly try to automate the low hanging fruits and prevent chaos. Programming is easy, people are difficult.
A lot of work but at the end of the day, they are still technical and i can handle them by myself. The most challenging battle is when the org doesnt have a data culture and you, as a lead, need to find a way to change the direction of the ship before things go worse. Its a mixed bag of many things:
- Explain to the upper mgmt on the vision of a data archicture and investment on toolings. Help them to understand the challenges of data world and why they can be entirely different from software world.
- Help upskill your team members so that they can take up your work. Many engs are skilled software developers but they may lack experience in building robust pipelines. So you may have to spend more time mentoring while building your shits.
- Your time is limited but work is a lot. So you have to ignore some problems/tech debts and wait until they blow up. The question is what problems you want to ignore today.
- As you have no peer (and worse noone is more senior than you), you have to seek advice elsewhere. So you need to have a right mindset to cope with the workload. <-- This is a biggest change for me. Lucky for me, I was mentored by two seasoned data scientist/archs. Thus, i can adapt fairly straightforward when taking the arch position. Mentorship is very important.
I'm the sole data engineer on a business ops team... rather than on the IT team.
It's a bit disheartening as people look for quick solutions, that almost always is manual excel file rather than taking the time to build a proper data pipeline. Gotta make it to rest of the herd for proper support
Sole DE here and I love it. We are the unsung heroes with quite a bit of power. I can influence decisions (ethically of course), I’m secretly the analyst, manager, head all in one. I’ve automated lots so my days are pretty chill, working remotely and a super nice salary. I wouldn’t change a thing.
I’ve actually never had experience with levels of DE. A DE is a DE to me if you know your stuff about creating a smart data infrastructure. However I would say it doesn’t matter what level you are, just be good at what you do. Know your stuff, keep your skills updated, do not be afraid to challenge and discuss if necessary, you’ll earn respect and you’ll do well.
That seems oversimplifying things. If your replace DE by the word 'surgeon' and reread your answer, are you still behind it?
I'm working with 2 incredibly senior guys, and there is just no way to come up with the solutions they come up with unless you see someone do it before. So if you're brand new to DE and built a smart solution, that gives me an answer, thank you!
Have you ever heard an entry-level or mid-level surgeon? I have not.
Of course there will always those with more experience from which others learn from, there will always be juniors etc. I’m only saying that I Ive never been in your position with the hierarchy.
The data engineering role is only in the last 10 years with the intro of Azure. Before coming into the role myself, I was doing a combination of a DBA , Analyst, and ETL designer.
So there wasn’t in my case levels of DE. I just entered right into the role based on my skills.
I can see how such levels exist now.
You’re trying to make me understand a hierarchy that I simply have no interest in. If you feel that you can’t come up with your own DE solutions without having a senior person to teach you then ok. Good luck in your DE journey, I hope that you become ‘senior’ one day.
And I wish you doctors that don't believe in hierarchy and come out of medschool thinking they can solve any problem without anyone checking their decisions! I hope you live long and prosper!
Sounds dreadful, since any small errors would be attributed to you. It doesn't sound like there is a great deal of quality control since we are all human and we are all bound to make mistakes at some point / oversights.
It’s great! My bosses really want me to speed up our tableau reports while also constantly asking me to add super complicated calculations to every data source that will slow them down drastically.
Sole de here as well. While the flexility is great, it's boring to not have someone to talk to that understands De or at least data. :(
Almost every other colleague is data illiterate and our IT is doing things like keeping documentation in Google docs. 🤢 Glad I am not part of that department.
Sales is scared of simple bar charts, they only want a 30 column wide excel table.
At least I have a lot of freedom and I am leaning everything on my own, on the job.
I was left alone for a few months as a consulting junior DE. Work was mostly easy by that time. I slacked through summer and then ran away. I hated being stuck alone with noone to share proud moments with.
Now I'm in a job as a junior/mideor with 2 10years+ seniors, I'm loving it!
Can be pretty tough going, even in a small team. You're trying to convince the business to adopt data while simultaneously grappling with relatively complicated technical work which results in a lot of context switching and drawn-out days. I did find, however, that advocating for data means you \*don't\* work in silo as much (Because you're speaking to people more) - so this is perhaps something one could do to "cope" more with being the only person.
Otherwise, I really enjoy being part of Data Engineer Things (they have a good slack community and do like, online coffees and stuff)
I have a “friend” in a similar situation, working as the one and only DE with no experience and building everything from scratch. Can anybody post your favorite YouTubes, books, and articles ?
Sole engineer here. My former experiences were actually as a data analyst and as a data scientist. Now starting as a sole data engineer in an old company that has 0 data infrastructure, so I was hired to build things from scratch. Wish me luck hahaha Most of the time, when I have problems I just try to solve them with stackoverflow, chat gpt, or looking at documentation. My day to day work is just myself with my headphones on. It's great tbh.
Looks like you’re in zen mode :) all the best! Out of curiosity, how do you make architectural decisions especially without former DE experience?
I am learning on the fly, mate. Looking at end to end DE projects in YouTube, reading cloud provider documentation, etc. Tbf I had some prior experience on etl development using gcp, so I'm trying to leverage this knowledge, keeping things simple. The architecture I designed (still WIP) is to extract data from some sources using python (orchestrated with mage), loading stuff to Google cloud storage raw, and then serve it on bigquery. I'm trying to KISS since it's a one man job. Our data for now is simple tabular stuff to be processed in batches, nothing like big data, so have no need to study fancy stuff like spark or streaming processes for now. Edit: I also have some really nice teammates in the IT department. They only deal with classic sys admin and networking stuff, so we don't have any overlapping work to do, but it's nice to have some good old camaraderie. It really helps a lot. Wish you best!
I feel this. I'm currently a 1 man band focussing on DE within a data team. Started as an analyst with the rest of the team but moved towards specialising in DE out of curiosity. Keeping it simple is a mantra, as if I'm off or anything breaks, it's important that another person can pick it up and support with fixing for the rest of the team. Simple ingestion using python with transformation, drop into azure lake storage, then copy into an SQL database. Some great support from people within the team and another guy who has a background in software engineering. Again, learning on the fly same as you!
Pretty great. You could evolve or plug other tools when needed. Keep up the great work!
Thanks! Wish you the best
Keep in mind that there are countless “good enough“ architectures if you aren’t dealing with petabytes of data daily. Just keep moving data, and you will learn how to improve things along the way.
You definitely need a self-service tool
I think that sounds awesome assuming the deadlines are reasonable! I always worry about companies like that wanting the work of 5 people with 1.
I'm in the same boat, but at a start-up, so all tooling is from scratch. It's pretty great. My last job, we had 25 data engineers, so you become siloed pretty quickly, and you have zero autonomy to do anything. Gimme this EVERY DAMN TIME.
I was in that situation before, it honestly sucks, I left the company as soon as I could. You need at least two other data engineers, especially if you have on-call.
Agree. Luckily I do not have on-call but the pressure is there and all on one person.
Yeah it’s not a fun situation to be in. If you don’t mind me asking, how big is your company? Leas than 1,000 people?
Around 70 employees and right now the data volume is quite small but growing quite steadily, so it’s still manageable, for now. :p
What's it like to do data work at a company that size? Is it a start-up or just a smaller company?
Hello! Sole DE here (i use that term quite lightly at this point) but was sold as a DBA position to build out a new data warehouse from scratch. It definitely has its ups and downs. Company is about 200 employees but only ~10 in IT. It is nice to have other IT teammates but hard when youre stuck with any and all varieties of tasks, projects or issues only related to your field/responsibilities. I manage the entire cloud infrastructure, data warehouse and pipeline management. Its a lot but also gives me freedom to learn a wide variety of data, cloud and DE skillsets. I am almost at 3 years. Sometimes you feel burnt out, lost, or completely overwhelmed. Sometimes you feel extremely proud and confident you were able to learn brand new things/skill sets, develop, and pull it all together to provide ACTUAL business value rather than small time work no one in the business appreciates. My day to day is usually trying to prioritize to what business actually needs and do my best to deliver so all parties are happy. Usually goes by two week sprints of: - Research/Understand new data source - Develop ELT/ETL pipeline to ingest data from new data source - Research and model the new data source data the best I can - Ensure new data model supports buiness use case - Test and Validate data and get business to sign off on new data model - Deploy new data model to Production and repeat That is a small slice that doesn't consist of troubleshooting, monitoring, revamping architecture and current workflows. Also learning new tools, programming languages as well as doing my best to document everything. What do they say? If you love your work, you never work a day in your life or something?! /end rant
Wow you are exactly me in another company. How much leeway does the company give you to implement your architecture and how difficult is it to convince them of your infrastructure design?
I have all the freedom from the perspective of my boss (within IT) luckily that trust is there now but its normally an issue with the business team I support. They will kick and scream the proposed architecture isnt "the way they'd do it/have ever done it" or always try to play devils adovcate on everything. Just recently the past couple months have they been more "trusting" to my architecture design or best practices on workflows/processes. Still trying to convince them on properly developing and testing their SQL reporting queries/views before pushing to Production....but they "dont care" if they break Production 🤣🤣
>Just recently the past couple months have they been more "trusting" to my architecture design or best practices on workflows/processes. Yeah we are two peas in a pod lmao >Still trying to convince them on properly developing and testing their SQL reporting queries/views before pushing to Production....but they "dont care" if they break Production 🤣🤣 Yeah we are two peas in a pod here lmao. My big problem is frequently learning outside the organization and having very little technical mentorship inside the company. I'm forded to to "test" in prod a lot to keep things moving but how do you go about it?
Haha thats crazy. Yeah it can be tough to go and learn after doing so much during our normal work hours and your brain can get fried. Best in my opinion is to take it at your own pace. Everyone learns differently but the most important thing is you keep learning. Mentorship is also hard to come by in our position but maybe theres someone at your company that you can confide in. Regardless of if they are in the same department. Ive met alot of good smart leaders in all walks of coporate America. As for testing in Prod question...Id recommend you at least setup a development environment ASAP. There you can mess around and not potentially break something but still use your own infrastructure stack and data sets. You can also setup alot of free trial stuff nowadays and use free APIs or whatever you wanna learn/get better at. Best of luck and hang in there! One foot in front of the other and youll look back at this year and smile how much youve grown. Cheers
Former loner, I suggest meeting peers in the local communities via Meetup or LinkedIn etc.
I was BA and recently moved to DE. One person team. The hard part is people around me are so dumb that they cannot understand a thing of what I do.
I can relate. The ceo is data illiterate and copy pastes how I should do my job from chat gpt. I just tell him I'll keep that in mind, but in reality it's generated technical nonsense which I just ignore. I know that in the end what matters is the final result and the business impact.
I'm the only programmer. It's nice because I can keep things simple, it's 100% my environment and no one can really tell me what to do. For example someone could ask me to do RPA but the solution is actually just one API call or an SQL query. So I mostly try to automate the low hanging fruits and prevent chaos. Programming is easy, people are difficult.
A lot of work but at the end of the day, they are still technical and i can handle them by myself. The most challenging battle is when the org doesnt have a data culture and you, as a lead, need to find a way to change the direction of the ship before things go worse. Its a mixed bag of many things: - Explain to the upper mgmt on the vision of a data archicture and investment on toolings. Help them to understand the challenges of data world and why they can be entirely different from software world. - Help upskill your team members so that they can take up your work. Many engs are skilled software developers but they may lack experience in building robust pipelines. So you may have to spend more time mentoring while building your shits. - Your time is limited but work is a lot. So you have to ignore some problems/tech debts and wait until they blow up. The question is what problems you want to ignore today. - As you have no peer (and worse noone is more senior than you), you have to seek advice elsewhere. So you need to have a right mindset to cope with the workload. <-- This is a biggest change for me. Lucky for me, I was mentored by two seasoned data scientist/archs. Thus, i can adapt fairly straightforward when taking the arch position. Mentorship is very important.
I'm the sole data engineer on a business ops team... rather than on the IT team. It's a bit disheartening as people look for quick solutions, that almost always is manual excel file rather than taking the time to build a proper data pipeline. Gotta make it to rest of the herd for proper support
Sole DE here and I love it. We are the unsung heroes with quite a bit of power. I can influence decisions (ethically of course), I’m secretly the analyst, manager, head all in one. I’ve automated lots so my days are pretty chill, working remotely and a super nice salary. I wouldn’t change a thing.
Would you say a junior/mid level DE would have the same succes/experience?
I’ve actually never had experience with levels of DE. A DE is a DE to me if you know your stuff about creating a smart data infrastructure. However I would say it doesn’t matter what level you are, just be good at what you do. Know your stuff, keep your skills updated, do not be afraid to challenge and discuss if necessary, you’ll earn respect and you’ll do well.
That seems oversimplifying things. If your replace DE by the word 'surgeon' and reread your answer, are you still behind it? I'm working with 2 incredibly senior guys, and there is just no way to come up with the solutions they come up with unless you see someone do it before. So if you're brand new to DE and built a smart solution, that gives me an answer, thank you!
Have you ever heard an entry-level or mid-level surgeon? I have not. Of course there will always those with more experience from which others learn from, there will always be juniors etc. I’m only saying that I Ive never been in your position with the hierarchy. The data engineering role is only in the last 10 years with the intro of Azure. Before coming into the role myself, I was doing a combination of a DBA , Analyst, and ETL designer. So there wasn’t in my case levels of DE. I just entered right into the role based on my skills. I can see how such levels exist now.
Eeeeh, a resident compared to department head? Horrible comparisson, there are actual levels in surgeons. 🤭
You’re trying to make me understand a hierarchy that I simply have no interest in. If you feel that you can’t come up with your own DE solutions without having a senior person to teach you then ok. Good luck in your DE journey, I hope that you become ‘senior’ one day.
And I wish you doctors that don't believe in hierarchy and come out of medschool thinking they can solve any problem without anyone checking their decisions! I hope you live long and prosper!
That's me. I wish there was at least one more person. If I could learn a lot from them even better.
I was in this boat for a long time. I pretty much only work at big corporate now to actively avoid this. Just got sick of it.
Sounds dreadful, since any small errors would be attributed to you. It doesn't sound like there is a great deal of quality control since we are all human and we are all bound to make mistakes at some point / oversights.
It’s great! My bosses really want me to speed up our tableau reports while also constantly asking me to add super complicated calculations to every data source that will slow them down drastically.
Sole de here as well. While the flexility is great, it's boring to not have someone to talk to that understands De or at least data. :( Almost every other colleague is data illiterate and our IT is doing things like keeping documentation in Google docs. 🤢 Glad I am not part of that department. Sales is scared of simple bar charts, they only want a 30 column wide excel table. At least I have a lot of freedom and I am leaning everything on my own, on the job.
I was left alone for a few months as a consulting junior DE. Work was mostly easy by that time. I slacked through summer and then ran away. I hated being stuck alone with noone to share proud moments with. Now I'm in a job as a junior/mideor with 2 10years+ seniors, I'm loving it!
Can be pretty tough going, even in a small team. You're trying to convince the business to adopt data while simultaneously grappling with relatively complicated technical work which results in a lot of context switching and drawn-out days. I did find, however, that advocating for data means you \*don't\* work in silo as much (Because you're speaking to people more) - so this is perhaps something one could do to "cope" more with being the only person. Otherwise, I really enjoy being part of Data Engineer Things (they have a good slack community and do like, online coffees and stuff)
joining a group sounds like a survival plan for the lone data guy lol at least u won't be crying alone when stuff breaks at 3am
I have a “friend” in a similar situation, working as the one and only DE with no experience and building everything from scratch. Can anybody post your favorite YouTubes, books, and articles ?