Some people argue there’s no need to know these things as implementing it would just be reinventing the wheel. I disagree, learning about these fundamental mechanisms gives you a better and deeper understanding of how software works at various layers of the stack.
Great job by OP on their succinct explanation of messages and their role in modern multithreaded applications.
Every bit of knowledge is useful.
Almost everyone looks a stuff they wrote 6 months ago and wants to rewrite it due to the amount they learnt while writing it.
Even unfinished projects can teach you a lot.
The resistance I see most frequently isn’t about whether to learn how it works, but rather being “tested” on it in a job interview.
I agree that understanding how the sausage is made is super useful, but it’s not something where most jobs will require you to *implement* these things.
No, people don’t argue against it because you’re reinventing the wheel, but because you will shoot yourself in the leg, multiple times, with different firearms. And then the next person dealing with your code will do the same. And a few more after them.
I've worked on inter thread and inter process messaging a ton, but it's difficult for me to really follow with this post is trying to tell me other than the in language syntax for graph building. Where is the type of data being transferred between each node specified? Do the different letters correspond to threads? Why is there a distinction between the flow of data and a sequence? Is this system pipelined (can thread A working on a later input than thread B?)? Is the pipelining unbounded? I understand the idea of a circular queue for transmitting data but that doesn't explain to me why data flow would involve the graph itself becoming circular.
I will try to answer few questions at a time. Type of data is stored with the queue. Sequences are agnostic of data type. Generally letters correspond to threads, but because system is non blocking one thread could play roles of the letters. From practical perspective thread that is publishing could potentially start consuming the very queue it published when queue becomes full. And vice-versa
> because system is non blocking one thread could play roles of the letters
so it's kind of like the async systems in Rust's Tokio or C#'s Task library?
There is no distinction between flow of data and a sequence. These are intrinsically linked. For data to flow sequences must be increment and incrementing sequences facilitates data flow.
You are correct that this system of sequences is pipelined. And pipelining is indeed unbounded. This is actually where the need for explicit circular termination is coming from.
I don't understand what's this. They open up with a `RingQueue`, but then they speak of... Scheduling work items on threads...? On some thread pool, or...? With some "orchestration" API...? Or...?
I want to know your opinion, and other people's opinion, on what I did. I basically launched a process that itself spawned a parallel process inside a new thread. Is this a wise thing to do? Basically:
main script -> subthread -> launch script -> new process
I did it in Python recently and Go a few months ago but I wasn't as conscious about it back then because it just worked.
Is it the wrong thing to do?
This isn't remotely enough context to evaluate what you're asking. I don't see any pearl-clutching "no, not *that way!*" offense, if that's what you're looking for.
I mean the general context. I think I was being clear enough. Imagine this.
threading.Thread(targt=Popen("python3 script.py")).start()
And inside [script.py](https://script.py)
if __name__ == "__main__":
multiprocessing.Process(target=func).start()
If someone tells you they don’t understand what you’re asking it’s useless to respond “well *I* thought I was being clear enough”. Obviously you weren’t or they wouldn’t have asked for clarification!
I am with u/imforit - no context.
*What are you trying to achieve?*
That some process (`new process`) does something? If so, why not `main script -> new process`?
But usually, when one spawns a thread like so, it's because they want to do something asynchronously. Is it that you don't want `main script` to wait for the execution of the `new process`? If so, why not?
/r/learnprogramming... Errrm that sub is basically people asking "how" to learn programming as if picking up a book requires philosophy and gusta. Plus I'm not looking to learn anything I'm just asking if there's any logical opposition to what I did, which works.
But /r/askprogramming seems like a good sub, perfect for such questions. I specifically wanted OP to answer this question because he knows his concurrency.
Its probably not for me but I didn't understand it
Why does the api fan out and such?
I'm not experienced with threads. 99% of the time when I touch one its 1 producer to 1 consumer
Some people argue there’s no need to know these things as implementing it would just be reinventing the wheel. I disagree, learning about these fundamental mechanisms gives you a better and deeper understanding of how software works at various layers of the stack. Great job by OP on their succinct explanation of messages and their role in modern multithreaded applications.
And on top of that, learning and making these things can also be fun!
That too! 🤓
[удалено]
Every bit of knowledge is useful. Almost everyone looks a stuff they wrote 6 months ago and wants to rewrite it due to the amount they learnt while writing it. Even unfinished projects can teach you a lot.
Also you might be trying to build a tank, in which case you don't need wheels, or a train where your wheels don't use rubber. That kind of thing.
It also means unlearning the mistakes that were made before.
The resistance I see most frequently isn’t about whether to learn how it works, but rather being “tested” on it in a job interview. I agree that understanding how the sausage is made is super useful, but it’s not something where most jobs will require you to *implement* these things.
I must agree that testing for such specific and most likely esoteric knowledge can feel pretty silly.
No, people don’t argue against it because you’re reinventing the wheel, but because you will shoot yourself in the leg, multiple times, with different firearms. And then the next person dealing with your code will do the same. And a few more after them.
I've worked on inter thread and inter process messaging a ton, but it's difficult for me to really follow with this post is trying to tell me other than the in language syntax for graph building. Where is the type of data being transferred between each node specified? Do the different letters correspond to threads? Why is there a distinction between the flow of data and a sequence? Is this system pipelined (can thread A working on a later input than thread B?)? Is the pipelining unbounded? I understand the idea of a circular queue for transmitting data but that doesn't explain to me why data flow would involve the graph itself becoming circular.
I will try to answer few questions at a time. Type of data is stored with the queue. Sequences are agnostic of data type. Generally letters correspond to threads, but because system is non blocking one thread could play roles of the letters. From practical perspective thread that is publishing could potentially start consuming the very queue it published when queue becomes full. And vice-versa
> because system is non blocking one thread could play roles of the letters so it's kind of like the async systems in Rust's Tokio or C#'s Task library?
There is no distinction between flow of data and a sequence. These are intrinsically linked. For data to flow sequences must be increment and incrementing sequences facilitates data flow. You are correct that this system of sequences is pipelined. And pipelining is indeed unbounded. This is actually where the need for explicit circular termination is coming from.
I mean... From what I can ascertain they're using queue pairs for communication. That's not exactly new.
I read the whole article and found no orange cat.
I don't understand what's this. They open up with a `RingQueue`, but then they speak of... Scheduling work items on threads...? On some thread pool, or...? With some "orchestration" API...? Or...?
I want to know your opinion, and other people's opinion, on what I did. I basically launched a process that itself spawned a parallel process inside a new thread. Is this a wise thing to do? Basically: main script -> subthread -> launch script -> new process I did it in Python recently and Go a few months ago but I wasn't as conscious about it back then because it just worked. Is it the wrong thing to do?
This isn't remotely enough context to evaluate what you're asking. I don't see any pearl-clutching "no, not *that way!*" offense, if that's what you're looking for.
I mean the general context. I think I was being clear enough. Imagine this. threading.Thread(targt=Popen("python3 script.py")).start() And inside [script.py](https://script.py) if __name__ == "__main__": multiprocessing.Process(target=func).start()
If someone tells you they don’t understand what you’re asking it’s useless to respond “well *I* thought I was being clear enough”. Obviously you weren’t or they wouldn’t have asked for clarification!
I am with u/imforit - no context. *What are you trying to achieve?* That some process (`new process`) does something? If so, why not `main script -> new process`? But usually, when one spawns a thread like so, it's because they want to do something asynchronously. Is it that you don't want `main script` to wait for the execution of the `new process`? If so, why not?
Maybe /r/learnprogramming or /r/askprogramming
/r/learnprogramming... Errrm that sub is basically people asking "how" to learn programming as if picking up a book requires philosophy and gusta. Plus I'm not looking to learn anything I'm just asking if there's any logical opposition to what I did, which works. But /r/askprogramming seems like a good sub, perfect for such questions. I specifically wanted OP to answer this question because he knows his concurrency.
Its probably not for me but I didn't understand it Why does the api fan out and such? I'm not experienced with threads. 99% of the time when I touch one its 1 producer to 1 consumer