Sure! I’ll do a proper write up regarding the problem of AI learning to do machine /assembly coding. I am not worried of computers getting to know coding for web development or proprietary software but hardware coding is a very different area that maybe only aligned AGI should be allowed to touch.
Yeah have seen copilot and yeah will try that definitely. It’s amazing and terrifying to see AI knowing how to code C and C++.
(why would assembly be extra problematic? Most languages turn into assembly in the end if they run on a CPU (even if after many stages), so why does it matter?)
Anyway, I’m betting OpenAI will get AI to help them invent better ML models, it might already be happening, and it will surely (in my opinion) snowball
It’s when the software can write assembly that can do something important without the benefit of existing libraries or an existing language (for example, C), that’s a very general capability, one that would help the software infer how to accomplish goals without the structure or boundaries of typical human uses of computers. It could be more creative than we’d like. That creativity would help an AGI planning to break out of an air-gapped computer system, for example.
“able to write capable code without using existing libraries”—yeah, that shows capabilities.
Doing that specifically in C and not in python? Doesn’t worry me as much. If it would happen in python (without using libraries), wouldn’t that concern you in a similar amount?
Hm, well, with C you can take advantage of hardware and coding errors a bit more easily, use memory management to do some buggy stuff, but with something like assembly you’re closer to working with the core hardware features, maybe taking advantage of features of the hardware design, finding and using CPU bugs, for example to take over management features, using side effects of hardware operation, doing things that might actually be harder to do in C than in assembly, because the compiler would get in the way.
I vaguely recall a discussion in Bostrom’s Superintelligence about software that used side effects of hardware function to turn motherboards without wifi into radios or something, I forget the details, but a language compiler tends to be platform independent or compensate for the hardware’s deficiencies, an AI that could write assembly wouldn’t want that..., hardware idiosyncrasies of the platform would be an advantage to it, it would want to be closer to the machine to find and use those for whatever purposes.
And again, knowing assembly at that level would show capabilities greater than knowing C.
I think an intelligent form that can think, has the capacity to understand and recreate itself is terrifying. That is why I am not supporting general artificial intelligence knowing how to code low level languages. At least javascript or python needs a compiler to become an assembly code and talk to the hardware. I hope I’m not missing something here. If yes please let me know.
I expect AI to be able to rewrite itself by writing python (pytorch?) code. Why wouldn’t that be enough? It was originally written by humans in python (probably)
Sure! I’ll do a proper write up regarding the problem of AI learning to do machine /assembly coding. I am not worried of computers getting to know coding for web development or proprietary software but hardware coding is a very different area that maybe only aligned AGI should be allowed to touch.
Yeah have seen copilot and yeah will try that definitely. It’s amazing and terrifying to see AI knowing how to code C and C++.
(why would assembly be extra problematic? Most languages turn into assembly in the end if they run on a CPU (even if after many stages), so why does it matter?)
Anyway, I’m betting OpenAI will get AI to help them invent better ML models, it might already be happening, and it will surely (in my opinion) snowball
It’s when the software can write assembly that can do something important without the benefit of existing libraries or an existing language (for example, C), that’s a very general capability, one that would help the software infer how to accomplish goals without the structure or boundaries of typical human uses of computers. It could be more creative than we’d like. That creativity would help an AGI planning to break out of an air-gapped computer system, for example.
“able to write capable code without using existing libraries”—yeah, that shows capabilities.
Doing that specifically in C and not in python? Doesn’t worry me as much. If it would happen in python (without using libraries), wouldn’t that concern you in a similar amount?
Hm, well, with C you can take advantage of hardware and coding errors a bit more easily, use memory management to do some buggy stuff, but with something like assembly you’re closer to working with the core hardware features, maybe taking advantage of features of the hardware design, finding and using CPU bugs, for example to take over management features, using side effects of hardware operation, doing things that might actually be harder to do in C than in assembly, because the compiler would get in the way.
I vaguely recall a discussion in Bostrom’s Superintelligence about software that used side effects of hardware function to turn motherboards without wifi into radios or something, I forget the details, but a language compiler tends to be platform independent or compensate for the hardware’s deficiencies, an AI that could write assembly wouldn’t want that..., hardware idiosyncrasies of the platform would be an advantage to it, it would want to be closer to the machine to find and use those for whatever purposes.
And again, knowing assembly at that level would show capabilities greater than knowing C.
I think an intelligent form that can think, has the capacity to understand and recreate itself is terrifying. That is why I am not supporting general artificial intelligence knowing how to code low level languages. At least javascript or python needs a compiler to become an assembly code and talk to the hardware. I hope I’m not missing something here. If yes please let me know.
I expect AI to be able to rewrite itself by writing python (pytorch?) code. Why wouldn’t that be enough? It was originally written by humans in python (probably)