It’s when the software can write assembly that can do something important without the benefit of existing libraries or an existing language (for example, C), that’s a very general capability, one that would help the software infer how to accomplish goals without the structure or boundaries of typical human uses of computers. It could be more creative than we’d like. That creativity would help an AGI planning to break out of an air-gapped computer system, for example.
“able to write capable code without using existing libraries”—yeah, that shows capabilities.
Doing that specifically in C and not in python? Doesn’t worry me as much. If it would happen in python (without using libraries), wouldn’t that concern you in a similar amount?
Hm, well, with C you can take advantage of hardware and coding errors a bit more easily, use memory management to do some buggy stuff, but with something like assembly you’re closer to working with the core hardware features, maybe taking advantage of features of the hardware design, finding and using CPU bugs, for example to take over management features, using side effects of hardware operation, doing things that might actually be harder to do in C than in assembly, because the compiler would get in the way.
I vaguely recall a discussion in Bostrom’s Superintelligence about software that used side effects of hardware function to turn motherboards without wifi into radios or something, I forget the details, but a language compiler tends to be platform independent or compensate for the hardware’s deficiencies, an AI that could write assembly wouldn’t want that..., hardware idiosyncrasies of the platform would be an advantage to it, it would want to be closer to the machine to find and use those for whatever purposes.
And again, knowing assembly at that level would show capabilities greater than knowing C.
It’s when the software can write assembly that can do something important without the benefit of existing libraries or an existing language (for example, C), that’s a very general capability, one that would help the software infer how to accomplish goals without the structure or boundaries of typical human uses of computers. It could be more creative than we’d like. That creativity would help an AGI planning to break out of an air-gapped computer system, for example.
“able to write capable code without using existing libraries”—yeah, that shows capabilities.
Doing that specifically in C and not in python? Doesn’t worry me as much. If it would happen in python (without using libraries), wouldn’t that concern you in a similar amount?
Hm, well, with C you can take advantage of hardware and coding errors a bit more easily, use memory management to do some buggy stuff, but with something like assembly you’re closer to working with the core hardware features, maybe taking advantage of features of the hardware design, finding and using CPU bugs, for example to take over management features, using side effects of hardware operation, doing things that might actually be harder to do in C than in assembly, because the compiler would get in the way.
I vaguely recall a discussion in Bostrom’s Superintelligence about software that used side effects of hardware function to turn motherboards without wifi into radios or something, I forget the details, but a language compiler tends to be platform independent or compensate for the hardware’s deficiencies, an AI that could write assembly wouldn’t want that..., hardware idiosyncrasies of the platform would be an advantage to it, it would want to be closer to the machine to find and use those for whatever purposes.
And again, knowing assembly at that level would show capabilities greater than knowing C.