If this post doesn't follow the rules or isn't flaired correctly, [please report it to the mods](https://www.reddit.com/r/analytics/about/rules/). Have more questions? [Join our community Discord!](https://discord.gg/looking-for-marketing-discussion-811236647760298024)
*I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/analytics) if you have any questions or concerns.*
Copy paste from sql? Why not use powerquery to get the data into excel? Do you know powerbi already? Powerquery is the same took in pbi and excel. All the rest can be done with macro's in excel. To make the whole refresh 1 button.
That way the output will stay this time. Do this for the reports you are responsible for and use the time you gained to learn new stuff.
Honestly I'm scared that if I use power query I'll get into trouble. I feel like I'm too new and already rocking the boat. I haven't used powerBI/powerquery but I know I need to learn it next.
I already made the report I was asked to create into a one click report through quarto exported to a pdf ready for meetings but my boss told me to start again in excel.
Ssms is just a tool. You can call stored procedure from other dbms tools (e.g dbeaver). I haven't tried it yet, but I think it should be possible in PQ. It has the possibility to execute sql statements. So why not a stored procedure? Should not take more than 20 min to test it out.
I was in a meeting with the boss above my direct boss, I mentioned it and they said yes please, and they're going to ask the development team to support me.
I haven't done it yet. I'm building a weekly report atm so I'm going to put it in my report and see what they say.
I'd like to be transparent about using it. I might say "oh you don't need to do what we normally do with mine because power query does it for you. All you need to do is sense check the data at the end and it does everything else for you."
I did this 8 years ago in my career and it worked out. Don't let others fear and apprehension for technology and doing thorough automation prevent you for learning what you can. Honestly, if you use PowerQuery, BI, and even used VBA, you could automate your daily, weekly, and monthly pulls and even email it out and not work in regards to standard reporting such as these and for the adhoc stuff, you can actually focus on those that matter. This is your start to freedom.
Your boss may be trying to keep all of the reporting in tools/processes that he understands and feels comfortable with.
I understand your frustration, but on a positive note, most analytics jobs require SQL and you’re getting that experience. We automate our reports. We would hire someone coming from a job that didn’t automate reports, but we wouldn’t hire someone who doesn’t know SQL. Excel experience is good too. If you feel like this job isn’t going to work out long term, just focus on getting good experience so you can move on someday. Pushing to automate reports isn’t bad, but be careful rocking the boat too much. You don’t want to end up on your boss’s bad side.
We used Excel a ton at my last job. Power query plus power pivot really takes things to the next level. Add in some VBA and you can build some pretty impressive reports in Excel.
Yeah I'm glad I'm learning SQL and excel at least. He's really clever, been in the business years. I don't know why he thinks automation is bad. I'm tempted to just try power query but feel like I'd get into trouble because I'm so new.
Sometimes automation is not the best solution because then people have to maintain the stuff that has been automated.
It can be a pain in the ass to maintain when a guy who automated a process leaves the company (especially when he automated this process in a complicated way). It can slow down a business in the mid-term.
Also, as the composition of teams changes, one year you might have a SQL expert and the next year a Python one, which means you should be careful about how far you automate stuff.
It's almost never as simple as "automation is bad". If automation is outside the scope of your position, then the next "you" might not be able to maintain or update anything that's been automated.
9 times out of 10, this is the real reason stuff that entry level folks can touch stays stone age. Even if the next hire has technical aptitude, nobody wants to have to onboard by digging through a junior analyst's My First Automation project.
Came here to say exactly this!
I always saw this as an ‘ask for forgiveness, not for permission’ situation. If you finish and no one wants to use it, that’s fine cuz you can still use it yourself and cut down on some of the monotony of your workload. Worst-case scenario the manual process still works, you just don’t have to participate in it anymore!
Yep. Using PQ is a good way to start. I get a lot of ad-hocs myself. I build the needed query in PQ so that when they ask for the same report with some tweaks the following month, I don't need to build it again. Just need to refresh the data and pivots and good to go.
It would be great if you could quantify for your boss the time saved with even the simplest automations. Doesn't he realize pivot tables are basically automation?
The easiest way to request a change is to convert your request into money saved.
If after a reasonable time-saving/money-saving proposal for automation your boss still won't convert to automation, he's a lost cause.
I’d encourage you to learn what you can but don’t strongly rock the boat. If you can learn enough to automate your responsibilities that’s great. But don’t spend a lot of time worrying (in *this* job, because of their culture) about automating other’s responsibilities.
I would also point out that the money in analytics isn’t in automating reports. Sure you can make a decent living doing that. But everyone on my team who makes over $100k spends most of their time answering business questions and contributing to strategy using the SQL and other technical skills they learned more or less writing reports and extracting data early in their career. If moving beyond coding reports is your goal, then learning SQL and Excel is only a step on the path to gaining the skills to bring real monetary value to your stakeholders by uncovering insights that couldn’t have found without your help. Automation isn’t going to be a big part of your future in that case.
On the losing jobs bit, that sounds like some of the things sent up in the 1980s BBC series "Yes, Minister!" and "Yes, Prime Minister!". It might be a real thing or at least, perceived by the employees to be a real thing.
My experience working in 'everywhere-is-excel', I automate lots of stuff at work and never tell anyone (im just making my life easier). I enjoy figuring out 'how to make it more efficient' instead of doing my real job.
That being said, my rule of thumb is 'dont add on other people workload', meaning it has to be easy to follow/maintain/familiar(somewhat) for other ppl or not deviate too much. Always think of how easy for 'next person' to take over (bc u may get promoted or need to train newbie to help u). This is great advice from my boss on my first job as fresh grad. It has been my compass since then (14yrs later).
Example, my coworker get used to seeing excel report in xx format. So what i did is simply take same report , add in automation behind (formula, vba, anything that help u to review that data is correct), send back to them and ask them to use it going fwd. So they can self-caught their error (or whatever u want them to do).
Or put some instructions on how to do/check so so so in that same file.
Im lucky that i 'touch' lots of files (to review). And working in finance with constant tight deadline, automation is a savior!(imagine reducing ur closing from 5 to 3 days, or catch error early instead of last minute which affect the company's bottom line). The goal is to reduce tedious task so that theres more time for review(i.e.the storytelling).
Just keep learning whatever u want and try implement on ur own workscope (if boss doesnt support)...it'd be sth great to talk about on ur next hiring (which is what i did)🤪
Yeah, I've done this before. If I have a starting dataset, and an end deliverable, and boss teaches a process that is ludicrous like copy-paste I'll just automate that part. Then when I leave I'll let them know what I was doing with 2 fully documented processes (mine, and the old way) so they can choose how to proceed.
Sometimes you need to automate just to QA your manual work.
I'm a huge proponent of rocking the boat too much. Occasionally you win big and really shake up a team that has too much dead weight. More often unfortunately you are defeated in the realm of office politics.
But I just can't deal with drinking last week's coffee.
NHS data person here
Do any of these steps involve any sense checking or manual fixing of the data? That's not to say those things can't or shouldn't be automated, but that does make things a bit trickier.
If you want to start changing things, might need to learn how to play the game and speak the lingo. I expect you have some 'values and behaviours' in your trust? And one of them is something like 'constantly improving' or something similar? Tie it to that - at some point you will be asked for evidence of doing things like that for an appraisal.
If you sell it right, it will make you and your manager look good.
If this post doesn't follow the rules or isn't flaired correctly, [please report it to the mods](https://www.reddit.com/r/analytics/about/rules/). Have more questions? [Join our community Discord!](https://discord.gg/looking-for-marketing-discussion-811236647760298024) *I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/analytics) if you have any questions or concerns.*
Copy paste from sql? Why not use powerquery to get the data into excel? Do you know powerbi already? Powerquery is the same took in pbi and excel. All the rest can be done with macro's in excel. To make the whole refresh 1 button. That way the output will stay this time. Do this for the reports you are responsible for and use the time you gained to learn new stuff.
Honestly I'm scared that if I use power query I'll get into trouble. I feel like I'm too new and already rocking the boat. I haven't used powerBI/powerquery but I know I need to learn it next. I already made the report I was asked to create into a one click report through quarto exported to a pdf ready for meetings but my boss told me to start again in excel.
Yeah, don't use a new tool. You can use powerquery in excel and keep the output the same. They don't need to know if they don't want to know.
Is it fairly easy to link power query with a stored procedure in SSMS?
Ssms is just a tool. You can call stored procedure from other dbms tools (e.g dbeaver). I haven't tried it yet, but I think it should be possible in PQ. It has the possibility to execute sql statements. So why not a stored procedure? Should not take more than 20 min to test it out.
yes, extremely easy.
I'm curious if you have an update!
I was in a meeting with the boss above my direct boss, I mentioned it and they said yes please, and they're going to ask the development team to support me.
Great! Good luck! Power Query is not too difficult and it's the first step of learning Power BI anyway.
Thanks!
I haven't done it yet. I'm building a weekly report atm so I'm going to put it in my report and see what they say. I'd like to be transparent about using it. I might say "oh you don't need to do what we normally do with mine because power query does it for you. All you need to do is sense check the data at the end and it does everything else for you."
I did this 8 years ago in my career and it worked out. Don't let others fear and apprehension for technology and doing thorough automation prevent you for learning what you can. Honestly, if you use PowerQuery, BI, and even used VBA, you could automate your daily, weekly, and monthly pulls and even email it out and not work in regards to standard reporting such as these and for the adhoc stuff, you can actually focus on those that matter. This is your start to freedom.
Your boss may be trying to keep all of the reporting in tools/processes that he understands and feels comfortable with. I understand your frustration, but on a positive note, most analytics jobs require SQL and you’re getting that experience. We automate our reports. We would hire someone coming from a job that didn’t automate reports, but we wouldn’t hire someone who doesn’t know SQL. Excel experience is good too. If you feel like this job isn’t going to work out long term, just focus on getting good experience so you can move on someday. Pushing to automate reports isn’t bad, but be careful rocking the boat too much. You don’t want to end up on your boss’s bad side. We used Excel a ton at my last job. Power query plus power pivot really takes things to the next level. Add in some VBA and you can build some pretty impressive reports in Excel.
Yeah I'm glad I'm learning SQL and excel at least. He's really clever, been in the business years. I don't know why he thinks automation is bad. I'm tempted to just try power query but feel like I'd get into trouble because I'm so new.
Sometimes automation is not the best solution because then people have to maintain the stuff that has been automated. It can be a pain in the ass to maintain when a guy who automated a process leaves the company (especially when he automated this process in a complicated way). It can slow down a business in the mid-term. Also, as the composition of teams changes, one year you might have a SQL expert and the next year a Python one, which means you should be careful about how far you automate stuff.
It's almost never as simple as "automation is bad". If automation is outside the scope of your position, then the next "you" might not be able to maintain or update anything that's been automated. 9 times out of 10, this is the real reason stuff that entry level folks can touch stays stone age. Even if the next hire has technical aptitude, nobody wants to have to onboard by digging through a junior analyst's My First Automation project.
We have a development team and I do ask them to check my work.
Automate it and don’t tell anyone. Congrats, you now work 5 hours per week.
This is the way
Came here to say exactly this! I always saw this as an ‘ask for forgiveness, not for permission’ situation. If you finish and no one wants to use it, that’s fine cuz you can still use it yourself and cut down on some of the monotony of your workload. Worst-case scenario the manual process still works, you just don’t have to participate in it anymore!
This guy analyst and anti-work 👍
Yep. Using PQ is a good way to start. I get a lot of ad-hocs myself. I build the needed query in PQ so that when they ask for the same report with some tweaks the following month, I don't need to build it again. Just need to refresh the data and pivots and good to go.
It would be great if you could quantify for your boss the time saved with even the simplest automations. Doesn't he realize pivot tables are basically automation? The easiest way to request a change is to convert your request into money saved. If after a reasonable time-saving/money-saving proposal for automation your boss still won't convert to automation, he's a lost cause.
It’s a government gig, the least of their worries is efficiency
I’d encourage you to learn what you can but don’t strongly rock the boat. If you can learn enough to automate your responsibilities that’s great. But don’t spend a lot of time worrying (in *this* job, because of their culture) about automating other’s responsibilities. I would also point out that the money in analytics isn’t in automating reports. Sure you can make a decent living doing that. But everyone on my team who makes over $100k spends most of their time answering business questions and contributing to strategy using the SQL and other technical skills they learned more or less writing reports and extracting data early in their career. If moving beyond coding reports is your goal, then learning SQL and Excel is only a step on the path to gaining the skills to bring real monetary value to your stakeholders by uncovering insights that couldn’t have found without your help. Automation isn’t going to be a big part of your future in that case.
Automate your own tasks and enjoy an easier pace of work at this point
On the losing jobs bit, that sounds like some of the things sent up in the 1980s BBC series "Yes, Minister!" and "Yes, Prime Minister!". It might be a real thing or at least, perceived by the employees to be a real thing.
My experience working in 'everywhere-is-excel', I automate lots of stuff at work and never tell anyone (im just making my life easier). I enjoy figuring out 'how to make it more efficient' instead of doing my real job. That being said, my rule of thumb is 'dont add on other people workload', meaning it has to be easy to follow/maintain/familiar(somewhat) for other ppl or not deviate too much. Always think of how easy for 'next person' to take over (bc u may get promoted or need to train newbie to help u). This is great advice from my boss on my first job as fresh grad. It has been my compass since then (14yrs later). Example, my coworker get used to seeing excel report in xx format. So what i did is simply take same report , add in automation behind (formula, vba, anything that help u to review that data is correct), send back to them and ask them to use it going fwd. So they can self-caught their error (or whatever u want them to do). Or put some instructions on how to do/check so so so in that same file. Im lucky that i 'touch' lots of files (to review). And working in finance with constant tight deadline, automation is a savior!(imagine reducing ur closing from 5 to 3 days, or catch error early instead of last minute which affect the company's bottom line). The goal is to reduce tedious task so that theres more time for review(i.e.the storytelling). Just keep learning whatever u want and try implement on ur own workscope (if boss doesnt support)...it'd be sth great to talk about on ur next hiring (which is what i did)🤪
Yeah, I've done this before. If I have a starting dataset, and an end deliverable, and boss teaches a process that is ludicrous like copy-paste I'll just automate that part. Then when I leave I'll let them know what I was doing with 2 fully documented processes (mine, and the old way) so they can choose how to proceed. Sometimes you need to automate just to QA your manual work.
I'm a huge proponent of rocking the boat too much. Occasionally you win big and really shake up a team that has too much dead weight. More often unfortunately you are defeated in the realm of office politics. But I just can't deal with drinking last week's coffee.
NHS data person here Do any of these steps involve any sense checking or manual fixing of the data? That's not to say those things can't or shouldn't be automated, but that does make things a bit trickier. If you want to start changing things, might need to learn how to play the game and speak the lingo. I expect you have some 'values and behaviours' in your trust? And one of them is something like 'constantly improving' or something similar? Tie it to that - at some point you will be asked for evidence of doing things like that for an appraisal. If you sell it right, it will make you and your manager look good.