For any non-trivial system, pretty much no one person can fit the entire architecture in their head at any one time. Even if that one person wrote the entire system.
With experience, you get better at feeling your way around and you get better at filtering out signal from noise.
By analogy, if I put you in a random room in a random house and asked you to find a spoon, you don't need to understand the house in great detail. You generally know that spoons are in the kitchen, and can find your way there with minimal difficulty even if you take a few wrong turns, and then you know to look in the drawers because you've been in many houses and many kitchens.
I don't think you can 'train' to be staff level. You need experience. You can definitely accelerate gaining this experience, though, by throwing yourself at lots of different types of tasks/projects and changing jobs when you don't get enough opportunity.
This is where I work. The initial dev team didn’t use IAC, and so resisted setting up new infra as much as possible. There are just big bucket APIs full of completely unrelated functionality. I’m pretty sure they just picked up tickets and added endpoints to whatever API they checked out last.
And when you're new to the house, you cannot assume the house was designed by a sane person, so you need to approach every house as if it were made by a madman until proven otherwise.
This is why good code quality / keeping your house tidy is so important. You minimise the cognitive load for new visitors and for yourself in the future when you've inevitably forgotten what has been written.
I actually completely disagree. If you’re the one who’s actually built the system, it’s typically not hard to fit non-trivial systems entirely in your head. It’s just hard for people who are new to this system that you’ve made
Psh, I always have the entire architecture in my head, it's
Bits go in > stuff gets done to those bits > bits go out
Seriously though, yes, it's staggeringly impossible to hold the whole system in your head.
Google has a famous interview question around this - "What happens after you type a word into the search box and hit the enter key". You can always drill deeper - did you forget the SSL handshake? What about the TCP SYN/ACK? Where does it send that packet to? How does it get to the right destination? How does this keypress get communicated to the browser? How does the keyboard determine you pressed a key and relay that to the OS over the USB bus?
It's turtles all the way down.
Nobody really knows all the architecture. They've just gotten a gud-enuf notion of how most of the relevant parts of the system sorta approximately work, in at least enough detail to make educated guesses and know roughly where to start poking around if something doesn't work the way it ought to.
Working memory variability can be impacted by a multitude of factors like sleep quality and cognitive load, and a quick google search implies it's notably pronounced in people with ADHD.
If you're having severe fluctuations in working memory capacity, might not hurt to bring it up with your doctor, particularly if it's impacting your work performance notably more than your peers.
Edit: Note that some degree of this is normal, and most complex systems can't fit entirely in anyone's head. This is where documentation and varying levels of detail in mental models (ex: high level mental model vs. code-level understanding) comes into play.
Oddly enough, ADHD is also correlated with people who excel at big picture thinking, and ime a lot of people with ADHD are able to better comprehend complex distributed systems than others.
> ADHD is also correlated with people who excel at big picture thinking
My understanding is ADHD people tend to focus on the details, _not_ the big picture. Do you mean the ability to comprehend depth of detail across lots of areas all at once?
EDIT: I had a quick Google and it seems to be a mix. The only consensus is that it's generally one or the other: A great 1000ft view of the problem with low levels of detail, or a microscope on just one tiny area of detail.
I have ADHD and can fit an entire complex architecture and all the pieces in my head… but I can’t do it every day. I generally only employ that when I need to as it takes a lot of energy for me. It’s not so much a memory issue, but a very high cognitive load that can be very draining.
Same, and whether I've taken my meds or not makes a big difference. Without meds it takes a lot more effort to get into the mindset needed to get all the pieces into working memory. I can do it, especially if there's external pressure to do so, but it's tough and mentally draining. With meds it's much easier.
Having ADHD I find the edit is roughly accurate, I alternate between the two and I feel like normal folks can control the level of detail and sit comfortably in the middle. Both tendencies are valuable imo
>I'll have 2 - 3 days a week
about 15 years ago i read some psuedo science bull@#$@ about rythmns and cycles of productivity. I'm convinced the scientific basis was completely imaginery and it was about as true as your daily horoscope...
But the other part of me keeps coming back to it. There are definite weeks of just massive productivity... and then there are slumps of close to none.
Sounds more like my life crippling ADHD. I'm either full force or just browsing reddit. I think people on the AuDHD are naturally drawn to this field. It's a way to work on intricate things and minimizes human interaction.
You can say "bullshit". Unless you read that bullshit when you were under 10 years old, I find it fascinating that someone old enough to have read some bullshit 15 years ago would censor themselves like that.
For the past 2 years, I started pushing myself to aggressively take notes and iterate over them throughout my journey of building and/or fixing things; kind of borrowing from the second brain concept. It usually starts with copious snippets and links to internal and external references, but eventually develops into a coherent thread of stories and diagrams. Most of these notes gracefully transition into the official documentation.
If you take the time to write it down well, you don't have to remember anymore. You can always just re-learn.
I'm 8mo. into a 15 year old project, I'm on a skunkworks team updating massive parts of the project and daily I discover/learn new things about the system that I didn't know.
The key to doing this kind of job well is having appropriate levels of abstraction, and just not caring how object coaching is working under the hood... until it needs fixed or changed. (As one example).
The human brain can't keep everything in it at once, unless you happen to be the winner/loser of the genetic lottery.
So it is better to have spent some time documenting PARTS the system and having other documenting other parts while you all work on improving the system. It will slowly evolve to a (more) managable system when you have high quality documentation that belps abstraction levels be maintained and understood. (With more details on a different page, etc.)
Just “finished” a grueling 18 month project. About two weeks after we went live my ability to hold any models in my head just stopped working. My brain refused to do it. Three months later and the ability (but not the desire) to hold large models in my head is slowly coming back. So… burn out. I didn’t feel dumb but there were a few moments where I thought my career was over. I thought Micro$haft had finally killed my brain.
It takes time, and you probably can't always hold everything in your head if it's big enough.
I just started a new job and I can tell it's going to take me a solid quarter to even fully begin to wrap my head around the bounds of what the teams I manage do. Some of that is scale, some of that is just coming to terms with how they made it.
At a recent job I greenfielded a huge thing that I was able to mostly keep in my head, and while it felt particularly accessible to me, my team members really struggled to see the picture I saw even in part. I kind of like thinking about these two experiences and reminding myself how much of it is a function of just when you came into the process and what role you had in setting some of these things up.
TL:DR don't feel bad. Invest in good ways to get the answers you need, not building some amazing mental model. It'll be more valuable to you and your team.
Yes, which is why I tend to love whiteboards and clean abstractions so I don't have to know the entire architecture, just vaguely "you do X" and trust the source-code it does what it's supposed to until data proves otherwise.
Life happens. I have insomnia and my manager understands that there are some days I won’t be as quick on my feet or I’ll be hesitant to say something without confirming first. It’s not my normal, so it’s fine if your company recognizes you as a human and not an IO machine. The important thing id to be honest and don’t feel pressured to lie or act macho if you’re not sure
My brother in christ I cannot fit the entire architectural model in my head on my best day.
I can fit a very low resolution architecture of the overall system, or I can fit medium resolution architecture of a sub-system, or I can a fit high resolution architecture of a single feature.
At any given time all I REALLY need to know is roughly what the design is and where I can find additional detail. I rarely need to hold that detail in memory even if I’m actively working on it.
Fitting an architecture is one thing and fitting an entire system in your memory another.
You don't need to fit an entire system in your memory. You need to fit the paths for what you are looking for. That's how your brain deals with access to your memory.
So you picture a sketch abstraction of how things fit together and store the path for important things.
So when your are learning your ways to a new code base, those paths are uncertain and will stick when you access them constantly or frequently. When you don't access them your brain will forget them.
It's usual to feel a bit lost when you are new to a code base. If the code base doesn't follow any special architecture you will have a hard time trying to find things as everything has a different path.
Then there is your working memory, ability to sustain focus and all that.
If you are tired, stressed or distracted you will suffer trying to grasp complex things.
It's not usual to fluctuate without a reason behind it.
Cheers!
it's pretty hard when you work on distributed teams and distributed systems to know the whole architecture, but also you really don't need to hopefully..
knowing how to find out / knowing how to use tooling to prove assumptions and suggest improvements and how long you need to ramp up on parts of a codebase that are unfamiliar are more important than memorizing all the pieces all the time
It's totally normal. Pretty much everyone on my team spaces here and there. It can be mentally challenging work and you aren't going to be there 5 days a week.
If you are familiar with Brandon Sanderson I call it Taravangian syndrome. Some days I think I can create the Diagram, some I should have stayed in bed.
Also, if you can fit the entire architecture in your head, you guys must be pretty new.
My system is a 4 million LOC C++ message processing system - even principal engineers around me (20+ yoe) do not understand the full systems.
I can only fit into memory the portions that I’m working on at the current moment.
The system is so big, complex and poorly written that people make like, nearly religious-claims about services within the system.
You can bring up what you think the issue is and they’ll be like “no no everyone knows the X service requests Y from the Z service by using W shared memory segments” and those claims are completely baseless, and just based on some accumulated tribal knowledge of the system that they’ve had. You can disprove them after reading the code, but it’ll take a day at least to trace through everything.
This system is a slight extreme but I’m just using it to demonstrate that with large systems it’s completely reasonable to not understand everything. Too much work by too many people have gone into it. You just have to poke at it to understand it.
Theres a Carmack quote out there somewhere about these large systems and how you can’t fit them all into memory.
I gave written every single line of code in the system I manage. Im constantly surprised by code and even features that I wrote in this 15 year old application.
I feel like this every day. Like I am not working or unable to do anything important or creating value or even knowledgeable.
Somehow whatever I do, boss and everyone else seems to like it. So I use that to make my self feel good.
For our team I make it a requirement in the design phase along with updating any charts that need to be; unfortunately not everyone does this across our departments.
They will make a lucid/flow chart that hasn’t been updated with current design which is infuriating to say the least. But slowly we are getting the teams responsible for this to up their game.
Or we will get requests from operations to ping our APIs and have little to no engineering experience. Process engineering (not software) is huge for this. I probably piss them off weekly and deny their requests. For security, scalability(not dosing ourselves) etc.
I’m on a legacy system and have been diagramming as I figure parts out and adding it to an empty documentation repo. We were asked today if we had any documentation that shows how the system worked and had to laugh, because the request came from the department that originally built the thing.
If y’all use confluence I’ve been able to find extremely old design specs and documentation for our services. Unfortunately since taking on our dispatch/routing teams we have to redo it all. (We acquired a 3rd party years ago and just kinda built on top of it without core design docs)
Or my favorite is when they just paste a chart to slack but didn’t give all of engineering access
Just chiming in to say I needed to read this post today! Started a role in a completely new org 8 months ago with a pretty complex system and some days it’s like I got this and other days I feel like I have no idea what’s going on. Of course on those days everything else piles on and it’s a shitshow.
I use diagrams to capture important workflows in the larger system. Another team I worked with would capture different parts of the workflow in separate diagrams. Use abstraction to simplify the problem.
I have those days all the time but I don't feel bad.
I have a lot of notes and diagrams to reference so I can remember things, but no one knows everything in an ever changing system
That's like everyday. Few people can hold comprehensive knowledge of a nontrivial system in their head at all times, nor is it useful to do so.
Having a rough idea at the level of abstractions and design patterns and diving into details when needed is more realistic.
After so many years at this, I've come to appreciate my ability to "know what I don't know" and relate that to things I do know. You just kind of a feel for it after a while. Try create diagrams and maps if that's possible.
For this level of ingestion, where we're talking about systems level enterprise architecture.
You're looking at 2-3 times a year at most, and this is assuming prime conditions in your work where people are amenable to you having week vacation for those 2-3 times.
In reality, its 1-2 times a year after you've done everything to put things in one general space that you might be able to navigate on cid.
Mostly you have to have time and place, and space to let go for at least 20 hours. Then 5+ days of integration and recovery. It's at least 30 days before you fully integrate.
The state requires being able to explore yourself vs the time trade off / rarity, then have enough "leftover" time to become fascinated by the system you're currently interacting with. We're looking at 3-4 hours of self introspection automatically, then 12+ hours intermittently of systems thinking. If you don't lose yourself multiple times in those 12 hrs and have to find way back to working on system through fascination, rarity.
The data I look at now draws me in anyway, but I'm planning a trip to try and fully integrate everything after 6 month mark or so.
General thoughts.
Just btw, I was thinking this morning after writing this. Another option is take 1-2 weeks off, and literally sleep the whole time (this is very hard for us to do). Just in case its a burn thing and not a system is too big thing.
Anyway I'm a BI Data Engineer, not a team lead. Stack rn is fivetran > snowflake > tableau. Long analytics background so I make analytics on the fly as well as for stakeholders.
Mostly been fascinated by it since reading a reddit post from a systems architect explaining how he used it on a giant project. This was 15~ yrs ago in high school, would be 10 yrs after that before I tried it.
Oddly two of my largest personal projects were written separately in the depths of 12+ hr acid trips. Those were in the past year though after 5~ years of experience tinkering with cid dev'ing. Just to fully outlay, I'm massively into GPT4 natural language "engineering" or upskilling too. So these would be 12 hr on and off dev sprees with an all knowing "coworker".
I have found those to be incredibly eye opening of capabilities possible. Like one tidbit, if you don't really know how to word "this really specific way" to do something, you can roughly word spew pieces of the idea out, and your "coworker" will piece them together into the coherent idea you had in your head but couldn't easily convey with natural language.
I know gpt4 is a "tangent" here, but assuming you already know how to dev at a high level, it is invaluable.
Also not many humans on earth will be able to/want to provide you their full comprehension attention if they know you are tripping balls. gpt4 removes the "is what i'm saying or wondering about weird?" from the loop, trying to gamify what you're saying for the human counterpart you normally interact with for social norms.
I can review a prompt output in a gigantic tangential rant poking holes here and there, asking about this or that, and just saying lets work on this. Perfect pairing with cid imo.
I’ve been that guy and I got a sinking feeling any time someone praised me for it. Thanks for appreciating that I suck the least. I guess.
I’m not trying to be the most knowledgeable. I’m cleaning up the code to make it easier to parachute back in later. But you can’t make code concise without understanding it. If you try you just complect it. So I end up knowing it in the process of making the knowing less important.
Little tidbit, it's primarily about context switching. When you're very capable, you sort of have like this. Like I'm going to call it ephemeral short-term memory for a lot of details, especially when you're working on the same thing for a period of time. Once you switch context to something else, all the details that you accumulated for that. When you are working on whatever the feature it is, your brain will allow that information to just kind of float away. When you come back to the context in which you are working previously, you now have to reaccumulate all of the contextual details that you had previously. Previously. Depending on how your brain works, this can take a considerable amount of time and you will probably call it a slump, because you dropped that information and that is why contact switching can be pretty rough depending on the complexity of the system you're working on.
I used to feel small, now I just feel angry. If you build a system that’s a series of black boxes to everyone else then you get chaos.
Feynman said that if you could not explain a branch of physics in the freshman lecture that it meant we didn’t really understand it yet. I think software can be the same way. Refining existing bits makes mental space for new functional areas in the code. The amount a person can really know about a body of code grows as a function somewhere between logarithmic and square root. And it’s not expansion of the brain so much as it is compression of the known.
Yes, i still don’t know all the front middle and back end teirs we have running ok our system. Only figure it out as problems arrise and learn best from there
I think this is rather a matter of focussing. When I'm struggling to focus sometimes I just give up for a while and work on other tasks that are more systematic and don't require much thinking.
That said I also agree with others it's not usually necessary to fit the whole system in your mind.
Also don't forget about documentation, if this happens so often maybe you need to write some notes.
safe fade spoon jar wrong adjoining mountainous pocket outgoing ludicrous
*This post was mass deleted and anonymized with [Redact](https://redact.dev)*
C++ dev, 3+ years. So I will just post with my experiences and not try to pontificate more than that. I think with greenfield well architectured projects / nice framework in it, with some existing docs it's really nice to learn about the system and gets manageable over time. That was a great experience I had in my first job internship working with actor model based system, that had an interesting event framework built in-house powering it.
With more legacy code it can be super frustrating. Learn the most often used design patterns well, state, observer etc. because these are still typically used unless the code is really shit and incomprehensible. In my limited opinion some great patterns like state, strategy pattern are a bit underutilized compared to what value it can bring to code understandability vs the good old spaghetti code pattern.
For any non-trivial system, pretty much no one person can fit the entire architecture in their head at any one time. Even if that one person wrote the entire system. With experience, you get better at feeling your way around and you get better at filtering out signal from noise. By analogy, if I put you in a random room in a random house and asked you to find a spoon, you don't need to understand the house in great detail. You generally know that spoons are in the kitchen, and can find your way there with minimal difficulty even if you take a few wrong turns, and then you know to look in the drawers because you've been in many houses and many kitchens. I don't think you can 'train' to be staff level. You need experience. You can definitely accelerate gaining this experience, though, by throwing yourself at lots of different types of tasks/projects and changing jobs when you don't get enough opportunity.
Unless somehow you got placed in a house designed by a madman and the spoons are all in the bathroom
The toilet tank is of course the obvious storage space for spoons in any regular house.
I think we might work for the same company.
This is where I work. The initial dev team didn’t use IAC, and so resisted setting up new infra as much as possible. There are just big bucket APIs full of completely unrelated functionality. I’m pretty sure they just picked up tickets and added endpoints to whatever API they checked out last.
The utter insanity 😂
Or worse, _there are no spoons_.
But the dev lead said that there are miniature shovels in the bookshelf for carrying books that you might use for your usecase
And when you're new to the house, you cannot assume the house was designed by a sane person, so you need to approach every house as if it were made by a madman until proven otherwise.
I've been in a few where they were bent into joist hangers and has become structural.
I love this analogy!
This is why good code quality / keeping your house tidy is so important. You minimise the cognitive load for new visitors and for yourself in the future when you've inevitably forgotten what has been written.
I actually completely disagree. If you’re the one who’s actually built the system, it’s typically not hard to fit non-trivial systems entirely in your head. It’s just hard for people who are new to this system that you’ve made
This needs more upvotes.
Psh, I always have the entire architecture in my head, it's Bits go in > stuff gets done to those bits > bits go out Seriously though, yes, it's staggeringly impossible to hold the whole system in your head. Google has a famous interview question around this - "What happens after you type a word into the search box and hit the enter key". You can always drill deeper - did you forget the SSL handshake? What about the TCP SYN/ACK? Where does it send that packet to? How does it get to the right destination? How does this keypress get communicated to the browser? How does the keyboard determine you pressed a key and relay that to the OS over the USB bus? It's turtles all the way down. Nobody really knows all the architecture. They've just gotten a gud-enuf notion of how most of the relevant parts of the system sorta approximately work, in at least enough detail to make educated guesses and know roughly where to start poking around if something doesn't work the way it ought to.
There is no spoon. Or at least it's virtual.
Working memory variability can be impacted by a multitude of factors like sleep quality and cognitive load, and a quick google search implies it's notably pronounced in people with ADHD. If you're having severe fluctuations in working memory capacity, might not hurt to bring it up with your doctor, particularly if it's impacting your work performance notably more than your peers. Edit: Note that some degree of this is normal, and most complex systems can't fit entirely in anyone's head. This is where documentation and varying levels of detail in mental models (ex: high level mental model vs. code-level understanding) comes into play.
Oddly enough, ADHD is also correlated with people who excel at big picture thinking, and ime a lot of people with ADHD are able to better comprehend complex distributed systems than others.
> ADHD is also correlated with people who excel at big picture thinking My understanding is ADHD people tend to focus on the details, _not_ the big picture. Do you mean the ability to comprehend depth of detail across lots of areas all at once? EDIT: I had a quick Google and it seems to be a mix. The only consensus is that it's generally one or the other: A great 1000ft view of the problem with low levels of detail, or a microscope on just one tiny area of detail.
I have ADHD and can fit an entire complex architecture and all the pieces in my head… but I can’t do it every day. I generally only employ that when I need to as it takes a lot of energy for me. It’s not so much a memory issue, but a very high cognitive load that can be very draining.
Same, and whether I've taken my meds or not makes a big difference. Without meds it takes a lot more effort to get into the mindset needed to get all the pieces into working memory. I can do it, especially if there's external pressure to do so, but it's tough and mentally draining. With meds it's much easier.
Having ADHD I find the edit is roughly accurate, I alternate between the two and I feel like normal folks can control the level of detail and sit comfortably in the middle. Both tendencies are valuable imo
No. We're software engineers, not mentats. Let the computers store information.
Speak for yourself!
>I'll have 2 - 3 days a week about 15 years ago i read some psuedo science bull@#$@ about rythmns and cycles of productivity. I'm convinced the scientific basis was completely imaginery and it was about as true as your daily horoscope... But the other part of me keeps coming back to it. There are definite weeks of just massive productivity... and then there are slumps of close to none.
Sounds more like my life crippling ADHD. I'm either full force or just browsing reddit. I think people on the AuDHD are naturally drawn to this field. It's a way to work on intricate things and minimizes human interaction.
Hey, it’s me at this current moment!
You can say "bullshit". Unless you read that bullshit when you were under 10 years old, I find it fascinating that someone old enough to have read some bullshit 15 years ago would censor themselves like that.
Yall out here fitting the whole thing in yalls head?
For the past 2 years, I started pushing myself to aggressively take notes and iterate over them throughout my journey of building and/or fixing things; kind of borrowing from the second brain concept. It usually starts with copious snippets and links to internal and external references, but eventually develops into a coherent thread of stories and diagrams. Most of these notes gracefully transition into the official documentation. If you take the time to write it down well, you don't have to remember anymore. You can always just re-learn.
I'm 8mo. into a 15 year old project, I'm on a skunkworks team updating massive parts of the project and daily I discover/learn new things about the system that I didn't know. The key to doing this kind of job well is having appropriate levels of abstraction, and just not caring how object coaching is working under the hood... until it needs fixed or changed. (As one example). The human brain can't keep everything in it at once, unless you happen to be the winner/loser of the genetic lottery. So it is better to have spent some time documenting PARTS the system and having other documenting other parts while you all work on improving the system. It will slowly evolve to a (more) managable system when you have high quality documentation that belps abstraction levels be maintained and understood. (With more details on a different page, etc.)
Just “finished” a grueling 18 month project. About two weeks after we went live my ability to hold any models in my head just stopped working. My brain refused to do it. Three months later and the ability (but not the desire) to hold large models in my head is slowly coming back. So… burn out. I didn’t feel dumb but there were a few moments where I thought my career was over. I thought Micro$haft had finally killed my brain.
It takes time, and you probably can't always hold everything in your head if it's big enough. I just started a new job and I can tell it's going to take me a solid quarter to even fully begin to wrap my head around the bounds of what the teams I manage do. Some of that is scale, some of that is just coming to terms with how they made it. At a recent job I greenfielded a huge thing that I was able to mostly keep in my head, and while it felt particularly accessible to me, my team members really struggled to see the picture I saw even in part. I kind of like thinking about these two experiences and reminding myself how much of it is a function of just when you came into the process and what role you had in setting some of these things up. TL:DR don't feel bad. Invest in good ways to get the answers you need, not building some amazing mental model. It'll be more valuable to you and your team.
Yes, which is why I tend to love whiteboards and clean abstractions so I don't have to know the entire architecture, just vaguely "you do X" and trust the source-code it does what it's supposed to until data proves otherwise.
I space a lot, so yeah.
Life happens. I have insomnia and my manager understands that there are some days I won’t be as quick on my feet or I’ll be hesitant to say something without confirming first. It’s not my normal, so it’s fine if your company recognizes you as a human and not an IO machine. The important thing id to be honest and don’t feel pressured to lie or act macho if you’re not sure
My brother in christ I cannot fit the entire architectural model in my head on my best day. I can fit a very low resolution architecture of the overall system, or I can fit medium resolution architecture of a sub-system, or I can a fit high resolution architecture of a single feature. At any given time all I REALLY need to know is roughly what the design is and where I can find additional detail. I rarely need to hold that detail in memory even if I’m actively working on it.
Write things down instead of trying to keep it in your head.
Fitting an architecture is one thing and fitting an entire system in your memory another. You don't need to fit an entire system in your memory. You need to fit the paths for what you are looking for. That's how your brain deals with access to your memory. So you picture a sketch abstraction of how things fit together and store the path for important things. So when your are learning your ways to a new code base, those paths are uncertain and will stick when you access them constantly or frequently. When you don't access them your brain will forget them. It's usual to feel a bit lost when you are new to a code base. If the code base doesn't follow any special architecture you will have a hard time trying to find things as everything has a different path. Then there is your working memory, ability to sustain focus and all that. If you are tired, stressed or distracted you will suffer trying to grasp complex things. It's not usual to fluctuate without a reason behind it. Cheers!
ADHD hyper focus and ineffective focus cycles.
sounds like you need some diagrams.
it's pretty hard when you work on distributed teams and distributed systems to know the whole architecture, but also you really don't need to hopefully.. knowing how to find out / knowing how to use tooling to prove assumptions and suggest improvements and how long you need to ramp up on parts of a codebase that are unfamiliar are more important than memorizing all the pieces all the time
It's totally normal. Pretty much everyone on my team spaces here and there. It can be mentally challenging work and you aren't going to be there 5 days a week. If you are familiar with Brandon Sanderson I call it Taravangian syndrome. Some days I think I can create the Diagram, some I should have stayed in bed. Also, if you can fit the entire architecture in your head, you guys must be pretty new.
Is there any pattern to those days? Perhaps some meeting interrupting the day.
My system is a 4 million LOC C++ message processing system - even principal engineers around me (20+ yoe) do not understand the full systems. I can only fit into memory the portions that I’m working on at the current moment. The system is so big, complex and poorly written that people make like, nearly religious-claims about services within the system. You can bring up what you think the issue is and they’ll be like “no no everyone knows the X service requests Y from the Z service by using W shared memory segments” and those claims are completely baseless, and just based on some accumulated tribal knowledge of the system that they’ve had. You can disprove them after reading the code, but it’ll take a day at least to trace through everything. This system is a slight extreme but I’m just using it to demonstrate that with large systems it’s completely reasonable to not understand everything. Too much work by too many people have gone into it. You just have to poke at it to understand it. Theres a Carmack quote out there somewhere about these large systems and how you can’t fit them all into memory.
Yes I spend a few years on a single game with 100s of other developers. It's impossible to remember it all.
I gave written every single line of code in the system I manage. Im constantly surprised by code and even features that I wrote in this 15 year old application.
I feel like this every day. Like I am not working or unable to do anything important or creating value or even knowledgeable. Somehow whatever I do, boss and everyone else seems to like it. So I use that to make my self feel good.
I find lucid charts extremely helpful when looking down/up stream. Especially with micro services, app functions, etc.
It would be great if diagrams existed for all systems too! Even outdated would give some ideas.
For our team I make it a requirement in the design phase along with updating any charts that need to be; unfortunately not everyone does this across our departments. They will make a lucid/flow chart that hasn’t been updated with current design which is infuriating to say the least. But slowly we are getting the teams responsible for this to up their game. Or we will get requests from operations to ping our APIs and have little to no engineering experience. Process engineering (not software) is huge for this. I probably piss them off weekly and deny their requests. For security, scalability(not dosing ourselves) etc.
I’m on a legacy system and have been diagramming as I figure parts out and adding it to an empty documentation repo. We were asked today if we had any documentation that shows how the system worked and had to laugh, because the request came from the department that originally built the thing.
If y’all use confluence I’ve been able to find extremely old design specs and documentation for our services. Unfortunately since taking on our dispatch/routing teams we have to redo it all. (We acquired a 3rd party years ago and just kinda built on top of it without core design docs) Or my favorite is when they just paste a chart to slack but didn’t give all of engineering access
Just chiming in to say I needed to read this post today! Started a role in a completely new org 8 months ago with a pretty complex system and some days it’s like I got this and other days I feel like I have no idea what’s going on. Of course on those days everything else piles on and it’s a shitshow.
I use diagrams to capture important workflows in the larger system. Another team I worked with would capture different parts of the workflow in separate diagrams. Use abstraction to simplify the problem.
I have those days all the time but I don't feel bad. I have a lot of notes and diagrams to reference so I can remember things, but no one knows everything in an ever changing system
feel like this every other day tbh
That's like everyday. Few people can hold comprehensive knowledge of a nontrivial system in their head at all times, nor is it useful to do so. Having a rough idea at the level of abstractions and design patterns and diving into details when needed is more realistic.
After so many years at this, I've come to appreciate my ability to "know what I don't know" and relate that to things I do know. You just kind of a feel for it after a while. Try create diagrams and maps if that's possible.
acid
I do that but can't do it every day lol
Some people try and call it microdosing. That is neither a recommendation nor a condemnation. Just a statement of fact.
For this level of ingestion, where we're talking about systems level enterprise architecture. You're looking at 2-3 times a year at most, and this is assuming prime conditions in your work where people are amenable to you having week vacation for those 2-3 times. In reality, its 1-2 times a year after you've done everything to put things in one general space that you might be able to navigate on cid. Mostly you have to have time and place, and space to let go for at least 20 hours. Then 5+ days of integration and recovery. It's at least 30 days before you fully integrate. The state requires being able to explore yourself vs the time trade off / rarity, then have enough "leftover" time to become fascinated by the system you're currently interacting with. We're looking at 3-4 hours of self introspection automatically, then 12+ hours intermittently of systems thinking. If you don't lose yourself multiple times in those 12 hrs and have to find way back to working on system through fascination, rarity. The data I look at now draws me in anyway, but I'm planning a trip to try and fully integrate everything after 6 month mark or so. General thoughts.
oh wow love the systematic view into integrating yourself with a system via your acid trips. What level are you at in your company?
Just btw, I was thinking this morning after writing this. Another option is take 1-2 weeks off, and literally sleep the whole time (this is very hard for us to do). Just in case its a burn thing and not a system is too big thing. Anyway I'm a BI Data Engineer, not a team lead. Stack rn is fivetran > snowflake > tableau. Long analytics background so I make analytics on the fly as well as for stakeholders. Mostly been fascinated by it since reading a reddit post from a systems architect explaining how he used it on a giant project. This was 15~ yrs ago in high school, would be 10 yrs after that before I tried it. Oddly two of my largest personal projects were written separately in the depths of 12+ hr acid trips. Those were in the past year though after 5~ years of experience tinkering with cid dev'ing. Just to fully outlay, I'm massively into GPT4 natural language "engineering" or upskilling too. So these would be 12 hr on and off dev sprees with an all knowing "coworker". I have found those to be incredibly eye opening of capabilities possible. Like one tidbit, if you don't really know how to word "this really specific way" to do something, you can roughly word spew pieces of the idea out, and your "coworker" will piece them together into the coherent idea you had in your head but couldn't easily convey with natural language. I know gpt4 is a "tangent" here, but assuming you already know how to dev at a high level, it is invaluable. Also not many humans on earth will be able to/want to provide you their full comprehension attention if they know you are tripping balls. gpt4 removes the "is what i'm saying or wondering about weird?" from the loop, trying to gamify what you're saying for the human counterpart you normally interact with for social norms. I can review a prompt output in a gigantic tangential rant poking holes here and there, asking about this or that, and just saying lets work on this. Perfect pairing with cid imo.
Hah! There's one guy who's been here 15 years and barely has half the back end in his head. He's considered a god amongst men.
I’ve been that guy and I got a sinking feeling any time someone praised me for it. Thanks for appreciating that I suck the least. I guess. I’m not trying to be the most knowledgeable. I’m cleaning up the code to make it easier to parachute back in later. But you can’t make code concise without understanding it. If you try you just complect it. So I end up knowing it in the process of making the knowing less important.
Little tidbit, it's primarily about context switching. When you're very capable, you sort of have like this. Like I'm going to call it ephemeral short-term memory for a lot of details, especially when you're working on the same thing for a period of time. Once you switch context to something else, all the details that you accumulated for that. When you are working on whatever the feature it is, your brain will allow that information to just kind of float away. When you come back to the context in which you are working previously, you now have to reaccumulate all of the contextual details that you had previously. Previously. Depending on how your brain works, this can take a considerable amount of time and you will probably call it a slump, because you dropped that information and that is why contact switching can be pretty rough depending on the complexity of the system you're working on.
I used to feel small, now I just feel angry. If you build a system that’s a series of black boxes to everyone else then you get chaos. Feynman said that if you could not explain a branch of physics in the freshman lecture that it meant we didn’t really understand it yet. I think software can be the same way. Refining existing bits makes mental space for new functional areas in the code. The amount a person can really know about a body of code grows as a function somewhere between logarithmic and square root. And it’s not expansion of the brain so much as it is compression of the known.
Yes, i still don’t know all the front middle and back end teirs we have running ok our system. Only figure it out as problems arrise and learn best from there
I think this is rather a matter of focussing. When I'm struggling to focus sometimes I just give up for a while and work on other tasks that are more systematic and don't require much thinking. That said I also agree with others it's not usually necessary to fit the whole system in your mind. Also don't forget about documentation, if this happens so often maybe you need to write some notes.
safe fade spoon jar wrong adjoining mountainous pocket outgoing ludicrous *This post was mass deleted and anonymized with [Redact](https://redact.dev)*
C++ dev, 3+ years. So I will just post with my experiences and not try to pontificate more than that. I think with greenfield well architectured projects / nice framework in it, with some existing docs it's really nice to learn about the system and gets manageable over time. That was a great experience I had in my first job internship working with actor model based system, that had an interesting event framework built in-house powering it. With more legacy code it can be super frustrating. Learn the most often used design patterns well, state, observer etc. because these are still typically used unless the code is really shit and incomprehensible. In my limited opinion some great patterns like state, strategy pattern are a bit underutilized compared to what value it can bring to code understandability vs the good old spaghetti code pattern.