This newsgroup.

Post by Dan Cross
is anyone here actually working on developing real, novel
operating systems on modern hardware here?

Novel or not, an OS can't go far without device drivers. For that you
need manpower. I don't see much around here.

Dan Cross

2023-03-22 12:04:29 UTC

Post by Dan Cross
is anyone here actually working on developing real, novel
operating systems on modern hardware here?

Novel or not, an OS can't go far without device drivers. For that you
need manpower. I don't see much around here.

This is true. It strikes me that there are options here.

Virtualization with emulated devices is the obvious one.
While it doesn't eliminate it, this at least limits the scope
for drivers one must implement, if one is willing to use, say,
virtio for things like block storage, networking, etc.

Another option would be to use something like a rump kernel to
provide drivers for actual devices.

Porting drivers from existing systems is always an option,
though one that is a fair bit more labor intensive than the
others. Still, for a particularly complex device, it may be
less overall effort than building a new driver from scratch.

In many ways, we are lucky these days: there's far less
variation across bus architectures, storage options, etc, as
there once was.

- Dan C.

Scott Lurndal

2023-03-22 14:35:05 UTC

Post by Dan Cross
is anyone here actually working on developing real, novel
operating systems on modern hardware here?

Novel or not, an OS can't go far without device drivers. For that you
need manpower. I don't see much around here.

This is true. It strikes me that there are options here.
Virtualization with emulated devices is the obvious one.
While it doesn't eliminate it, this at least limits the scope
for drivers one must implement, if one is willing to use, say,
virtio for things like block storage, networking, etc.
Another option would be to use something like a rump kernel to
provide drivers for actual devices.
Porting drivers from existing systems is always an option,
though one that is a fair bit more labor intensive than the
others. Still, for a particularly complex device, it may be
less overall effort than building a new driver from scratch.
In many ways, we are lucky these days: there's far less
variation across bus architectures, storage options, etc, as
there once was.

There are standards for several OS <-> hardware interfaces,
such as UHCI/EHCI/XHCI for USB, AHCI for SATA, NVME for
PCIe SSDs; yet there are many, just as important that don't
have a standard hardware interface (I2C, I3C, eMMC/SD, SMI,
and particularly, network interfaces and hardware accelerators).

Dan Cross

2023-03-22 15:10:53 UTC

Post by Dan Cross
is anyone here actually working on developing real, novel
operating systems on modern hardware here?

Novel or not, an OS can't go far without device drivers. For that you
need manpower. I don't see much around here.

This is true. It strikes me that there are options here.
Virtualization with emulated devices is the obvious one.
While it doesn't eliminate it, this at least limits the scope
for drivers one must implement, if one is willing to use, say,
virtio for things like block storage, networking, etc.
Another option would be to use something like a rump kernel to
provide drivers for actual devices.
Porting drivers from existing systems is always an option,
though one that is a fair bit more labor intensive than the
others. Still, for a particularly complex device, it may be
less overall effort than building a new driver from scratch.
In many ways, we are lucky these days: there's far less
variation across bus architectures, storage options, etc, as
there once was.

Don't forget SPI, GPIO, memory controllers, and on and on. I'm
not saying it's a completely solved problem, but at least we
mostly have commonality in terms of a bus architecture for fast
perhipherals. MSI and MSI-X dramatically simplifies interrupt
routing, and so on.

I suspect the situation is far worse on user-oriented devices.

- Dan C.

James Harris

2023-12-12 21:27:40 UTC

Post by Dan Cross
is anyone here actually working on developing real, novel
operating systems on modern hardware here?

Novel or not, an OS can't go far without device drivers. For that you
need manpower. I don't see much around here.

This is true. It strikes me that there are options here.
Virtualization with emulated devices is the obvious one.
While it doesn't eliminate it, this at least limits the scope
for drivers one must implement, if one is willing to use, say,
virtio for things like block storage, networking, etc.
Another option would be to use something like a rump kernel to
provide drivers for actual devices.
Porting drivers from existing systems is always an option,
though one that is a fair bit more labor intensive than the
others. Still, for a particularly complex device, it may be
less overall effort than building a new driver from scratch.

Perhaps one's OS could provide an environment in which Linux device
drivers could be run, and the OS build process would be able to take
driver source code from the Linux source tree.

Both steps should happen automatically, AISI, so while there would
indeed be a bunch of work to get the system set up there would be almost
zero work to add each driver.

I should say I haven't looked in to the details so the amount of work
may be prohibitive but ATM it sounds like the most promising way to get
access to thousands of device drivers in one's own OS.

My personal preference would be to run 'foreign' device drivers in a
protected environment where they could not break anything in the kernel
by writing to a rogue pointer.

As it happens, I was looking at a video on writing a Linux device driver
earlier today. In case someone wants to know the basics, this sets out a
few options for a character-device driver:

--
James Harris

Dan Cross

2023-12-13 15:28:25 UTC

Post by Dan Cross
is anyone here actually working on developing real, novel
operating systems on modern hardware here?

Novel or not, an OS can't go far without device drivers. For that you
need manpower. I don't see much around here.

This is true. It strikes me that there are options here.
Virtualization with emulated devices is the obvious one.
While it doesn't eliminate it, this at least limits the scope
for drivers one must implement, if one is willing to use, say,
virtio for things like block storage, networking, etc.
Another option would be to use something like a rump kernel to
provide drivers for actual devices.
Porting drivers from existing systems is always an option,
though one that is a fair bit more labor intensive than the
others. Still, for a particularly complex device, it may be
less overall effort than building a new driver from scratch.

Perhaps one's OS could provide an environment in which Linux device
drivers could be run, and the OS build process would be able to take
driver source code from the Linux source tree.
Both steps should happen automatically, AISI, so while there would
indeed be a bunch of work to get the system set up there would be almost
zero work to add each driver.
I should say I haven't looked in to the details so the amount of work
may be prohibitive but ATM it sounds like the most promising way to get
access to thousands of device drivers in one's own OS.
My personal preference would be to run 'foreign' device drivers in a
protected environment where they could not break anything in the kernel
by writing to a rogue pointer.
As it happens, I was looking at a video on writing a Linux device driver
earlier today. In case someone wants to know the basics, this sets out a
http://youtu.be/pIUTaMKq0Xc

Yeah, that's hard for anything non-trivial. Linux intentionally
has little in the way of internal interfaces, so one finds that
drivers for complex devices quickly become entwined with much of
the rest of the kernel: memory and process management, storage
internals, and so forth. To bring in a Linux driver often means
providing shim interfaces for much of the Linux kernel, which
can force an overall "shape" onto the driver that one may not
want.

That said, I've ported drivers that are themselves very complex,
but where the overall interface to the rest of the kernel was
relatively small.

- Dan C.

Scott Lurndal

2023-12-13 15:59:33 UTC

Post by Dan Cross
is anyone here actually working on developing real, novel
operating systems on modern hardware here?

Novel or not, an OS can't go far without device drivers. For that you
need manpower. I don't see much around here.

This is true. It strikes me that there are options here.
Virtualization with emulated devices is the obvious one.
While it doesn't eliminate it, this at least limits the scope
for drivers one must implement, if one is willing to use, say,
virtio for things like block storage, networking, etc.
Another option would be to use something like a rump kernel to
provide drivers for actual devices.
Porting drivers from existing systems is always an option,
though one that is a fair bit more labor intensive than the
others. Still, for a particularly complex device, it may be
less overall effort than building a new driver from scratch.

Perhaps one's OS could provide an environment in which Linux device
drivers could be run, and the OS build process would be able to take
driver source code from the Linux source tree.
Both steps should happen automatically, AISI, so while there would
indeed be a bunch of work to get the system set up there would be almost
zero work to add each driver.
I should say I haven't looked in to the details so the amount of work
may be prohibitive but ATM it sounds like the most promising way to get
access to thousands of device drivers in one's own OS.
My personal preference would be to run 'foreign' device drivers in a
protected environment where they could not break anything in the kernel
by writing to a rogue pointer.
As it happens, I was looking at a video on writing a Linux device driver
earlier today. In case someone wants to know the basics, this sets out a
http://youtu.be/pIUTaMKq0Xc

Back in the day, the unix operating systems all had "driver
development kits" and defined driver API's.

For linux, people like Greg K-H absolutely refused to provide
any form driver development standard because that would allow
binary driver distribution - anathema to rabid open sourcers.

Which led to all kinds of bizarre attempts to adapt
third party binary drivers.

Post by Dan Cross
That said, I've ported drivers that are themselves very complex,
but where the overall interface to the rest of the kernel was
relatively small.

Yup. I wrote a optical jukebox driver for NT3.51 back in
the 90's. The interface to the kernel was well-defined.

Scott Lurndal

2023-03-22 01:55:44 UTC

Post by Dan Cross
So, is anyone here actually working on developing real, novel
operating systems on modern hardware here?

Been there, done that, moved on. Microkernels, distributed
operating systems, massively parallel systems, MCP, Unix,
Linux. All have common features, all have unique implementations.

I've sometimes considered that the golden age of operating system
design is behind us.

Dan Cross

2023-03-22 12:18:39 UTC

Post by Dan Cross
So, is anyone here actually working on developing real, novel
operating systems on modern hardware here?

Been there, done that, moved on. Microkernels, distributed
operating systems, massively parallel systems, MCP, Unix,
Linux. All have common features, all have unique implementations.
I've sometimes considered that the golden age of operating system
design is behind us.

I'm not sure I agree. We're largely stuck with system
interfaces that were first described in the late 1960s through
the early 1980s, but hardware has changed dramatically since
then. I don't think we've really stopped to ask ourselves
whether the systems we seem to have gravitated towards are still
a good fit for either the hardware we have, or the sorts of
programs we're running.

In particular, one sees lots of inate parallelism and
distribution across CPU, RAM, and IO fabrics at the hardware
level, implying asynchrony to use effectively. And one also
sees lots of asynchrony at the application layer, often embedded
in language runtimes and the like. But most systems provide a
highly synchronous system interface: so the asynchronous and
parallel application talks to the asychronous and parallel
hardware through this soda-straw of an interface designed to
make a multiprocessor machine with gigabytes of main memory,
a deep memory cache hierarchy, and an inherently asynchronous,
parallel IO architecture look like a PDP-11.

Interfaces like Linux's ioring implementation try to bridge this
gap within the context of an existing system, but are not
particularly orthogonal to the rest of the OS. One wonders if
we can do better, but it will require a shift in the accepted
wisdom about the system's overall architecture.

- Dan C.

T. Ment

2023-03-22 15:10:01 UTC

We're largely stuck with system interfaces that were first
described in the late 1960s through the early 1980s, but
hardware has changed dramatically since then.

Right, now we have bugs like Spectre.

Dan Cross

2023-03-22 15:14:01 UTC

We're largely stuck with system interfaces that were first
described in the late 1960s through the early 1980s, but
hardware has changed dramatically since then.

Right, now we have bugs like Spectre.

Yup. A system architecture that prefers all of physical memory
into the kernel's virtual address space doesn't make a lot of
sense anymore.

- Dan C.

wolfgang kern

2023-03-22 02:10:39 UTC

Post by Dan Cross
So, is anyone here actually working on developing real, novel
operating systems on modern hardware here? It would be
interesting to have a USENET outlet for such things, which I had
hoped this newsgroup might be, but it seems dominated by other
things.

me too like to see attempts to create an UEFI compliant OS from scratch
w/o reusing C-libraries nor copies from already existing Lonnix/M$.

My own OS retired with me (1985..2018). Contracts end Dec.2024.
I'm too old for a complete restart on new motherboards, they all are
UEFI only now :(
__
wolfgang

muta...@gmail.com

2023-03-22 05:53:05 UTC

me too like to see attempts to create an UEFI compliant OS from scratch
w/o reusing C-libraries nor copies from already existing Lonnix/M$.

What's wrong with using a C library to do the UEFI calls?

BFN. Paul.

wolfgang kern

2023-03-22 16:41:49 UTC

me too like to see attempts to create an UEFI compliant OS from scratch
w/o reusing C-libraries nor copies from already existing Lonnix/M$.

What's wrong with using a C library to do the UEFI calls?

easy answer: it is C, aka limited+bloated+weird HLL-stuff.
__
wolfgang

Dan Cross

2023-03-22 18:33:02 UTC

me too like to see attempts to create an UEFI compliant OS from scratch
w/o reusing C-libraries nor copies from already existing Lonnix/M$.

What's wrong with using a C library to do the UEFI calls?

easy answer: it is C, aka limited+bloated+weird HLL-stuff.

It certainly seems like the time for building large systems in C
has passed. Frankly, we have better choices now.

- Dan C.

Scott Lurndal

2023-03-22 18:55:18 UTC

me too like to see attempts to create an UEFI compliant OS from scratch
w/o reusing C-libraries nor copies from already existing Lonnix/M$.

What's wrong with using a C library to do the UEFI calls?

easy answer: it is C, aka limited+bloated+weird HLL-stuff.

It certainly seems like the time for building large systems in C
has passed. Frankly, we have better choices now.

We had better choices three decades ago when we were writing full
operating systems (large scale, distributed) in C++[*][**].

We designed a hypervisor at SGI in the late 90's in C++[**].

Or a modula-like language called Sprite which we used at burroughs
for the mainframe MCP (medium systems) or NEWP which the large systems
folks used for MCP development.

[*] Circa 1989, C++2.1.

[**] In both cases using a subset that eschewed features with significant
run-time overhead, such as RTTI and Exceptions (most of which showed
up in C++3.0). And avoiding SGIs standard C++ library.

Dan Cross

2023-03-22 19:08:09 UTC

Post by wolfgang kern
easy answer: it is C, aka limited+bloated+weird HLL-stuff.

It certainly seems like the time for building large systems in C
has passed. Frankly, we have better choices now.

We had better choices three decades ago when we were writing full
operating systems (large scale, distributed) in C++[*][**].

Eh.... I don't know that C++ is a particularly compelling
improvement over C. It retains many of C's poor semantics, such
as the type promotion rules that can lead to hidden UB,
nullable pointers, etc, while the object and template semantics
are often surprising and unintuitive.

Post by Scott Lurndal
We designed a hypervisor at SGI in the late 90's in C++[**].

Was that part of the Disco work?

Post by Scott Lurndal
Or a modula-like language called Sprite which we used at burroughs
for the mainframe MCP (medium systems) or NEWP which the large systems
folks used for MCP development.

For that matter, Modula-2, Modula-3 or Oberon. Wirth's efforts
are underappreciated in the space.

Post by Scott Lurndal
[*] Circa 1989, C++2.1.
[**] In both cases using a subset that eschewed features with significant
run-time overhead, such as RTTI and Exceptions (most of which showed
up in C++3.0). And avoiding SGIs standard C++ library.

I've been pleasantly surprised at how well Rust fits the problem
domain. It's not perfect, but is much better than I had assumed
it would be when I first started playing around with it and it's
miles ahead of C.

- Dan C.

Scott Lurndal

2023-03-22 22:23:58 UTC

Post by wolfgang kern
easy answer: it is C, aka limited+bloated+weird HLL-stuff.

It certainly seems like the time for building large systems in C
has passed. Frankly, we have better choices now.

We had better choices three decades ago when we were writing full
operating systems (large scale, distributed) in C++[*][**].

Post by Scott Lurndal
We designed a hypervisor at SGI in the late 90's in C++[**].

Was that part of the Disco work?

More like inspired by it. It was a skunkworks project with
about 8 people working on it, including the chief scientist jwag.

Post by Dan Cross
I've been pleasantly surprised at how well Rust fits the problem
domain. It's not perfect, but is much better than I had assumed
it would be when I first started playing around with it and it's
miles ahead of C.

It's on my list to look at, but probably not 'til I retire.

Dan Cross

2023-03-23 00:51:49 UTC

Post by wolfgang kern
easy answer: it is C, aka limited+bloated+weird HLL-stuff.

It certainly seems like the time for building large systems in C
has passed. Frankly, we have better choices now.

We had better choices three decades ago when we were writing full
operating systems (large scale, distributed) in C++[*][**].

Post by Scott Lurndal
We designed a hypervisor at SGI in the late 90's in C++[**].

Was that part of the Disco work?

More like inspired by it. It was a skunkworks project with
about 8 people working on it, including the chief scientist jwag.

Oh cool. I always thought that MIPS would be hard to virtualize
because of the ksegs and soft-TLBs. It was never clear to me
how a hypervisor could, in general, know the format of the guest
page tables. I know the Disco folks had to make some changes to
Irix to get it to work.

It's on my list to look at, but probably not 'til I retire.

Highly recommended. The first time I walked off the end of an
array in an ersatz ACPI parser and got a well-defined panic
instead of overwriting random data, the hair on my wrist stood
up.

- Dan C.

Scott Lurndal

2023-03-23 14:52:14 UTC

Post by wolfgang kern
easy answer: it is C, aka limited+bloated+weird HLL-stuff.

It certainly seems like the time for building large systems in C
has passed. Frankly, we have better choices now.

We had better choices three decades ago when we were writing full
operating systems (large scale, distributed) in C++[*][**].

Post by Scott Lurndal
We designed a hypervisor at SGI in the late 90's in C++[**].

Was that part of the Disco work?

More like inspired by it. It was a skunkworks project with
about 8 people working on it, including the chief scientist jwag.

Oh cool. I always thought that MIPS would be hard to virtualize
because of the ksegs and soft-TLBs.

Indeed, it took another decade and a half before MIPS added
virtualization support to the architecture, IIRC; Cavium
was also a MIPS shop (until 2012, when we started the switch
to ARM64) and we had looked at the MIPS virtualization extensions around
that time.

The SGI skunkworks project used a 2 processor HP Kayak with
Pentium II (IIRC) processors; we had an instance of windows
running alongside an instance of linux. Microsoft had
given us the source for NT4 - this was about the time
that SGI was looking to move the graphics workstations
to intel/microsoft based boxen.

Post by Dan Cross
It was never clear to me
how a hypervisor could, in general, know the format of the guest
page tables. I know the Disco folks had to make some changes to
Irix to get it to work.

When I was working on IRIX, I was not fond of either the software
managed TLB, coloring or the Kseg stuff; the MIPS project I worked on was called
Teak and was a distributed version of Irix (eventually cancelled)
for networks of R10k boxes.

Dan Cross

2023-03-23 19:49:31 UTC

Post by Dan Cross
Oh cool. I always thought that MIPS would be hard to virtualize
because of the ksegs and soft-TLBs.

Same with x86, of course: how long after VMWare did their
binary-rewriting thing before Intel introduced VMX?

Post by Scott Lurndal
The SGI skunkworks project used a 2 processor HP Kayak with
Pentium II (IIRC) processors; we had an instance of windows
running alongside an instance of linux. Microsoft had
given us the source for NT4 - this was about the time
that SGI was looking to move the graphics workstations
to intel/microsoft based boxen.

Ah, very interesting; this was roughly the same era as VMWare's
initial offerings: I'm guessing you pulled the same incrememntal
rewriting trick they did?

Post by Dan Cross
It was never clear to me
how a hypervisor could, in general, know the format of the guest
page tables. I know the Disco folks had to make some changes to
Irix to get it to work.

I get it from a hardware perspective: fewer transistors with a
software-managed TLB, but man...so many drawbacks.

- Dan C.

Scott Lurndal

2023-03-23 20:09:07 UTC

Post by Dan Cross
Oh cool. I always thought that MIPS would be hard to virtualize
because of the ksegs and soft-TLBs.

Same with x86, of course: how long after VMWare did their
binary-rewriting thing before Intel introduced VMX?

AMD implemented SVM before Intel did VMX. It was
generally available circa 2005, so a few years after VMware.

Ah, very interesting; this was roughly the same era as VMWare's
initial offerings: I'm guessing you pulled the same incrememntal
rewriting trick they did?

It's been a while now, but the initial take was more paravirtualization
a la xen.

Post by Dan Cross
It was never clear to me
how a hypervisor could, in general, know the format of the guest
page tables. I know the Disco folks had to make some changes to
Irix to get it to work.

I get it from a hardware perspective: fewer transistors with a
software-managed TLB, but man...so many drawbacks.

True. I'm quite grateful that we can now squeeze a hundred
billion transistors on a single SoC :-).

Dan Cross

2023-03-23 20:31:42 UTC

Post by Dan Cross
Oh cool. I always thought that MIPS would be hard to virtualize
because of the ksegs and soft-TLBs.

Same with x86, of course: how long after VMWare did their
binary-rewriting thing before Intel introduced VMX?

AMD implemented SVM before Intel did VMX. It was
generally available circa 2005, so a few years after VMware.

Other way around, wasn't it? Intel released VMX in late 2005
while AMD released SVM in mid 2006.

Ah, very interesting; this was roughly the same era as VMWare's
initial offerings: I'm guessing you pulled the same incrememntal
rewriting trick they did?

It's been a while now, but the initial take was more paravirtualization
a la xen.

That makes sense.

Post by Dan Cross
It was never clear to me
how a hypervisor could, in general, know the format of the guest
page tables. I know the Disco folks had to make some changes to
Irix to get it to work.

I get it from a hardware perspective: fewer transistors with a
software-managed TLB, but man...so many drawbacks.

True. I'm quite grateful that we can now squeeze a hundred
billion transistors on a single SoC :-).

All running at multi-gigahertz speeds. Quite the times we live
in.

- Dan C.

Scott Lurndal

2023-03-23 20:55:03 UTC

Post by Dan Cross
Oh cool. I always thought that MIPS would be hard to virtualize
because of the ksegs and soft-TLBs.

Same with x86, of course: how long after VMWare did their
binary-rewriting thing before Intel introduced VMX?

AMD implemented SVM before Intel did VMX. It was
generally available circa 2005, so a few years after VMware.

Other way around, wasn't it? Intel released VMX in late 2005
while AMD released SVM in mid 2006.

Yes, you're correct.

We had an early Pacifica (SVM) spec before it was released publically
in 2005.

I was thinking of the nested page table feature (which we relied
on heavily) which came before Intel's EPT. These were the features
that VMware and Xen really wanted to avoid the need to trap page
table updates by the guest (or paravirt them).

Dan Cross

2023-03-23 23:37:13 UTC

Post by Dan Cross
Oh cool. I always thought that MIPS would be hard to virtualize
because of the ksegs and soft-TLBs.

Same with x86, of course: how long after VMWare did their
binary-rewriting thing before Intel introduced VMX?

AMD implemented SVM before Intel did VMX. It was
generally available circa 2005, so a few years after VMware.

Other way around, wasn't it? Intel released VMX in late 2005
while AMD released SVM in mid 2006.

Yes, you're correct.
We had an early Pacifica (SVM) spec before it was released publically
in 2005.

Ah, cool.

Post by Scott Lurndal
I was thinking of the nested page table feature (which we relied
on heavily) which came before Intel's EPT. These were the features
that VMware and Xen really wanted to avoid the need to trap page
table updates by the guest (or paravirt them).

Ah, yes, absolutely. Shadow paging to multiplex the guest's
address space on the host virtual address space is not fun.

- Dan C.

Scott Lurndal

2023-03-24 14:21:09 UTC

Post by Dan Cross
Oh cool. I always thought that MIPS would be hard to virtualize
because of the ksegs and soft-TLBs.

Same with x86, of course: how long after VMWare did their
binary-rewriting thing before Intel introduced VMX?

AMD implemented SVM before Intel did VMX. It was
generally available circa 2005, so a few years after VMware.

Other way around, wasn't it? Intel released VMX in late 2005
while AMD released SVM in mid 2006.

Yes, you're correct.
We had an early Pacifica (SVM) spec before it was released publically
in 2005.

Ah, cool.

Ah, yes, absolutely. Shadow paging to multiplex the guest's
address space on the host virtual address space is not fun.

Indeed. For x86, xen had been relying on the segment limit
registers, which amd64 had eliminated. They later added
back support for xen (and perhaps vmware?) about the same
time they were developing pacifica.

James Harris

2023-12-12 21:42:00 UTC

...

Post by Dan Cross
It was never clear to me
how a hypervisor could, in general, know the format of the guest
page tables. I know the Disco folks had to make some changes to
Irix to get it to work.

I get it from a hardware perspective: fewer transistors with a
software-managed TLB, but man...so many drawbacks.

Handling a software-managed TLB may be more work, in a sense, but it
gives an OS developer more control, more feedback, more freedom, and
perhaps better opportunities for performance gains - as long as the TLB
is large enough.

Having the hardware carry out a walk of page tables (the only option if
the TLB can is updated by hardware) has long seemed to me like a bad
idea, and it doesn't scale very well as addresses get wider.

--
James Harris

Scott Lurndal

2023-12-12 23:56:53 UTC

Post by Dan Cross
It was never clear to me
how a hypervisor could, in general, know the format of the guest
page tables. I know the Disco folks had to make some changes to
Irix to get it to work.

I get it from a hardware perspective: fewer transistors with a
software-managed TLB, but man...so many drawbacks.

Handling a software-managed TLB may be more work, in a sense, but it
gives an OS developer more control, more feedback, more freedom, and
perhaps better opportunities for performance gains - as long as the TLB
is large enough.
Having the hardware carry out a walk of page tables (the only option if
the TLB can is updated by hardware) has long seemed to me like a bad
idea, and it doesn't scale very well as addresses get wider.

Having worked extensively with both models (SW: MIPS, HW: pretty much
every other single mass-produced microprocessor), there is, hands down,
no benefit to software table walks. Zero. Zilch. Don't even bother.

Hardware translation table walks scale rather well, and in modern
incarnations (e.g. ARMv8) are very flexible, supporting multiple
fundamental unit of translation sizes (e.g. 4k, 16k, 64k) and
higher level "huge pages". Add in the second level of walks
required for hardware VM guest page table walkers[*] and the software
walker becomes fragile and slow. The hardware walkers have
things like content addressible memory and intermediate translation
walk caches that software cannot do as effectively or efficiently.

[*] 22 distinct memory accesses to translate a guest VA using 4k pages for
both the guest and nested page tables.

BGB

2023-12-13 02:40:43 UTC

Post by Dan Cross
It was never clear to me
how a hypervisor could, in general, know the format of the guest
page tables. I know the Disco folks had to make some changes to
Irix to get it to work.

I get it from a hardware perspective: fewer transistors with a
software-managed TLB, but man...so many drawbacks.

Handling a software-managed TLB may be more work, in a sense, but it
gives an OS developer more control, more feedback, more freedom, and
perhaps better opportunities for performance gains - as long as the TLB
is large enough.
Having the hardware carry out a walk of page tables (the only option if
the TLB can is updated by hardware) has long seemed to me like a bad
idea, and it doesn't scale very well as addresses get wider.

Having worked extensively with both models (SW: MIPS, HW: pretty much
every other single mass-produced microprocessor), there is, hands down,
no benefit to software table walks. Zero. Zilch. Don't even bother.
Hardware translation table walks scale rather well, and in modern
incarnations (e.g. ARMv8) are very flexible, supporting multiple
fundamental unit of translation sizes (e.g. 4k, 16k, 64k) and
higher level "huge pages". Add in the second level of walks
required for hardware VM guest page table walkers[*] and the software
walker becomes fragile and slow. The hardware walkers have
things like content addressible memory and intermediate translation
walk caches that software cannot do as effectively or efficiently.
[*] 22 distinct memory accesses to translate a guest VA using 4k pages for
both the guest and nested page tables.

I will partly disagree:
Software TLB makes the hardware cheaper to implement (it raises an TLB
Miss exception and its job is done);
Also one can make the interrupt mechanism cheap (for hardware) as well
by treating it as a glorified branch-with-link (with the interrupt
handler needing to deal with everything else itself).

But, in terms of performance or ease-of-use, I will admit that hardware
page walks make this easier.

But, CPUs that are fast or not a pain to write OS level code for, are
not the only things that matter.

Claiming that there is no possible advantage to software TLB is like
arguing that there is no possible advantage to CPU's which only support
aligned memory access (and use slow emulation traps to deal with
unaligned access).

If there were no advantage, people wouldn't do it that way...

The main advantage of the flexibility of software TLB is that more
advanced features can be added without needing to pay for them in terms
of making the hardware more expensive (you merely pay for them in terms
of programmer effort and clock-cycles).

Then people can take the savings in one area, and spend them somewhere
else, say, like having a better FPU (or having an FPU at all), or maybe
even such extravagances as an integer-divide instruction.

But, all sort of a balancing act.

...

Scott Lurndal

2023-12-13 15:01:24 UTC

Post by BGB

Post by Dan Cross
It was never clear to me
how a hypervisor could, in general, know the format of the guest
page tables. I know the Disco folks had to make some changes to
Irix to get it to work.

I get it from a hardware perspective: fewer transistors with a
software-managed TLB, but man...so many drawbacks.

Handling a software-managed TLB may be more work, in a sense, but it
gives an OS developer more control, more feedback, more freedom, and
perhaps better opportunities for performance gains - as long as the TLB
is large enough.
Having the hardware carry out a walk of page tables (the only option if
the TLB can is updated by hardware) has long seemed to me like a bad
idea, and it doesn't scale very well as addresses get wider.

Having worked extensively with both models (SW: MIPS, HW: pretty much
every other single mass-produced microprocessor), there is, hands down,
no benefit to software table walks. Zero. Zilch. Don't even bother.
Hardware translation table walks scale rather well, and in modern
incarnations (e.g. ARMv8) are very flexible, supporting multiple
fundamental unit of translation sizes (e.g. 4k, 16k, 64k) and
higher level "huge pages". Add in the second level of walks
required for hardware VM guest page table walkers[*] and the software
walker becomes fragile and slow. The hardware walkers have
things like content addressible memory and intermediate translation
walk caches that software cannot do as effectively or efficiently.
[*] 22 distinct memory accesses to translate a guest VA using 4k pages for
both the guest and nested page tables.

Software TLB makes the hardware cheaper to implement (it raises an TLB
Miss exception and its job is done);

No, it really doesn't make hardware cheaper to implement. Not
since the late 80's, where the MMU chip was optional.

Post by BGB
Also one can make the interrupt mechanism cheap (for hardware) as well
by treating it as a glorified branch-with-link (with the interrupt
handler needing to deal with everything else itself).

That doesn't matter.

Unless you are developing for a consumer grade FGPA, the
hardware walker is far superior in every way.

Post by BGB
Claiming that there is no possible advantage to software TLB is like
arguing that there is no possible advantage to CPU's which only support
aligned memory access (and use slow emulation traps to deal with
unaligned access).

There is no advantage to software alignment handling.

Post by BGB
If there were no advantage, people wouldn't do it that way...

No new design in the last two decades has done that.

Post by BGB
The main advantage of the flexibility of software TLB is that more
advanced features can be added without needing to pay for them in terms
of making the hardware more expensive (you merely pay for them in terms
of programmer effort and clock-cycles).

At 3nm, there is more then enough area to lay down a dozen table
walkers (and most chips have one per core or hardware thread and
perhaps a dozen in the IOMMU. Heck, at 160nm there was plenty
of room. The engineering costs are in the noise.

And note, a software walker in an IOMMU would never have made
any sense whatsoever.

BGB

2023-12-13 19:28:41 UTC

Post by BGB

Post by Dan Cross
It was never clear to me
how a hypervisor could, in general, know the format of the guest
page tables. I know the Disco folks had to make some changes to
Irix to get it to work.

I get it from a hardware perspective: fewer transistors with a
software-managed TLB, but man...so many drawbacks.

Handling a software-managed TLB may be more work, in a sense, but it
gives an OS developer more control, more feedback, more freedom, and
perhaps better opportunities for performance gains - as long as the TLB
is large enough.
Having the hardware carry out a walk of page tables (the only option if
the TLB can is updated by hardware) has long seemed to me like a bad
idea, and it doesn't scale very well as addresses get wider.

Having worked extensively with both models (SW: MIPS, HW: pretty much
every other single mass-produced microprocessor), there is, hands down,
no benefit to software table walks. Zero. Zilch. Don't even bother.
Hardware translation table walks scale rather well, and in modern
incarnations (e.g. ARMv8) are very flexible, supporting multiple
fundamental unit of translation sizes (e.g. 4k, 16k, 64k) and
higher level "huge pages". Add in the second level of walks
required for hardware VM guest page table walkers[*] and the software
walker becomes fragile and slow. The hardware walkers have
things like content addressible memory and intermediate translation
walk caches that software cannot do as effectively or efficiently.
[*] 22 distinct memory accesses to translate a guest VA using 4k pages for
both the guest and nested page tables.

Software TLB makes the hardware cheaper to implement (it raises an TLB
Miss exception and its job is done);

No, it really doesn't make hardware cheaper to implement. Not
since the late 80's, where the MMU chip was optional.

Depends probably on scale here...

That doesn't matter.
Unless you are developing for a consumer grade FGPA, the
hardware walker is far superior in every way.

It makes a difference on a lot of the Spartan and Artix chips.
Particularly on the lower end, like XC7S25.

But, a lot of the boards for the XC7S25 don't even bother with things
like RAM modules, so one is hard-pressed to go too far outside of
microcontroller territory with these as a result.

Where, these are FPGA's with around 15K LUT.

IOW: Your RAM+ROM space is measured in kB; so may make sense, say, to go
with 48K of ROM and 16K of RAM, using the low 64K of address space as
the main memory map. One finding out (after buying it) that the 512K
QSPI RAM module only came on the board with an XC7A35T instead.

There are also the cheaper ICE40 FPGA's, but doing that much beyond an
8/16 microcontroller on an ICE40 is pushing it (the bigger end of this
range tends to be around 5K LUTs). But, say, one can implement something
similar to a 6502 or similar on this (and a few people managed to get
RV32I cores and similar to fit, though usually cutting some corners).

On the bigger and more expensive FPGA boards, RAM sizes are typically
measured in MB.

Though, one of the major FPGAs I was targeting was the comparably
massive XC7A100T, with around 63K LUTs.

The comparison between LUTs and transistors is a bit ambiguous, but a
"reasonable estimate" seems to be 1 LUT ~= 10 transistors.

So, say, 63K LUT ~= 630K transistors.
Though, there are a lot of "cheat" features here, like DSP48's and Block
RAM.

Going much bigger than this, Xilinx wants money to use Vivado, and the
licenses cost more than the dev-boards... (Say, managed to get a
XC7K325T board for around $100 off of AliExpress, but can't use it
because this isn't one of the free options in Vivado).

The bare FPGAs would be a fair bit cheaper than the dev-boards, but the
BGA packaging (and need for multi-layer PCBs) is a bit more of an ask
for hobbyist-level development (granted, KiKad and services like PCBWay
and similar exist, but this is still a bit of an ask).

Some of the smaller FPGA boards are available in form factors that can
fit into the larger DIP sockets (and others can be plugged on top of
wire-wrap perfboard or similar if needed via a pin-header interface).

There is no advantage to software alignment handling.

Besides making the L1 cache cheaper, for similar reasons.

I did spend this cost though, because it can make a big difference for
things like "memcpy()" and LZ decompression and similar.

Dealing with misaligned access does nearly double the size of the L1
cache though, for a direct-mapped design:
One may need to access two cache lines at a time, and check two
different addresses for hit/miss, rather than a single cache line and a
single address check.

But, on the positive side, it can make LZ decompression roughly 8x
faster on a 64-bit CPU (because, loading/storing values 1 byte at a
time, would be slow).

Though, this situation would have been even worse on a word-oriented
design (such as the DEC Alpha).

Post by BGB
If there were no advantage, people wouldn't do it that way...

No new design in the last two decades has done that.

SiFive is selling chips that take around 500 cycles whenever one makes a
misaligned access...

Meanwhile, what one can pull off an an Artix-7 or similar is mostly
limited to what would have been "state of the art" 30+ years ago.

Makes more sense to compare against what would have been available in
the early 1990s.

Granted, in terms of computational power vs price, one would be better
off with something like RasPi (in terms of raw computational speed, a
RasPi is going to run circles around a soft-processor implemented on an
FPGA).

A lot of other comparisons may be done relative to clock-speed, since if
the clock-speed is considered, there is no contest.

In some projects, it may make sense to use both a RasPi and an FPGA
board, letting the FPGA deal with things like hardware-level interfacing
and real-time tasks, and a RasPi for things like network access and
dealing with "application side" tasks.

In theory, Zynq boards also do this, but:
Getting Zynq boards to "actually work" is a bit harder to pull off;
The RasPi + Spartan/Artix route may still be cheaper.

Mostly this is because trying to pull off stable real-time stuff on the
RasPi *sucks* and also one is hard-pressed to drive the GPIO pins
reliably much outside of kHz territory (where FPGA has no issue driving
MHz logic on the IO pins).

A lot of the ASIC's also have 4 or 5 orders of magnitude more
transistors than the LUT equivalent in a typical FPGA...

And also roughly 2 orders of magnitude more clock-speed.

Say, the clock-speed delta between my custom CPU core and the Ryzen in
my PC is around 74x, with around 1000x more RAM, etc.

Though, in some cases, the measured performance delta has been less than
the 74x clock-speed delta.

Note, no "OoO" extravagance in this case either; the CPU is strictly
in-order.

Post by Scott Lurndal
And note, a software walker in an IOMMU would never have made
any sense whatsoever.

No IOMMU in my case.
No DMA controller or interrupt controller or similar either.

Nor PCIe (if PCIe were done at all, it would likely also be handled in
software; but can't readily plug PCIe cards into an FPGA dev-board, so
it doesn't really matter that much).

The USB interface was itself mostly software (the hardware interface
being presented along similar lines to a UART, but needing to do extra
stuff because "USB is weird", and effectively dealing with 9-bit bytes
in the FIFO). This would also be limited to roughly USB 1.1 speeds.

Pretty much all of the IO here is polling IO (apart from a clock
interrupt and a few other things).

Though, it is more a case of cutting corners wherever possible.
Say:
FPU natively only does ADD/SUB/MUL;
Did add FDIV, but it was shoe-horned through a fairly slow integer
divider (and takes roughly 120 clock cycles);
DAZ/FTZ semantics only (an FMA design that could have dealt more easily
with subnormal numbers, was also too expensive, and as-is the multiplier
only computes the high-order results internally).

There are FPU-SIMD ops, but:
They "barely" manage Binary32 precision;
They are also hard-wired for truncate-only rounding
Floating point rounding being, also expensive.

For a lot of tasks, DAZ/FTZ Truncate-Only SIMD ops surprisingly usable.
These being mostly pairs of Binary32 numbers in a 64-bit register, or 4
Binary16 numbers. For 128-bit SIMD, register pairs are used.

But, only really the larger FPGAs are big enough to handle the SIMD unit.

...

James Harris

2023-12-14 20:01:45 UTC

Post by Dan Cross
It was never clear to me
how a hypervisor could, in general, know the format of the guest
page tables. I know the Disco folks had to make some changes to
Irix to get it to work.

I get it from a hardware perspective: fewer transistors with a
software-managed TLB, but man...so many drawbacks.

Handling a software-managed TLB may be more work, in a sense, but it
gives an OS developer more control, more feedback, more freedom, and
perhaps better opportunities for performance gains - as long as the TLB
is large enough.
Having the hardware carry out a walk of page tables (the only option if
the TLB can is updated by hardware) has long seemed to me like a bad
idea, and it doesn't scale very well as addresses get wider.

On the contrary!

Brendan (BGB) has already covered hardware issues and I respect his
views and experience on that. The /software/ problem with forward page
tables is that while they are OK (but still unnecessary) for small
numbers of processes and concomitant address spaces they don't work so
well for larger systems.

Forward page tables take up a significant amount of RAM - as a rule of
thumb let's say up to 1/1000th of the address space, e.g. up to 4M for a
4G address space on x32. While that's no problem for a small number of
processes and the impact can be reduced in many cases it's still
unavoidably true that each time you add a new process you have to add a
new set of page tables. Have 5,000 processes and you need 5,000 sets of
forward page tables. The whole idea is, to use a technical term,
bonkers. As I said, the approach does not scale.

By contrast, with reverse page tables you need just ONE table per
machine, no matter how many processes you want to support. Instead of
taking 1/1000th of the address space for every process one would use,
say, 1/500th of the RAM for all processes combined.

I acknowledge that each process will need forward mappings but can
arrange to reserve them in runs of (first, last) addresses for much
greater space efficiency than you'd get with forward page tables.

Post by Scott Lurndal
Hardware translation table walks scale rather well, and in modern
incarnations (e.g. ARMv8) are very flexible, supporting multiple
fundamental unit of translation sizes (e.g. 4k, 16k, 64k) and
higher level "huge pages". Add in the second level of walks
required for hardware VM guest page table walkers[*] and the software
walker becomes fragile and slow. The hardware walkers have
things like content addressible memory and intermediate translation
walk caches that software cannot do as effectively or efficiently.

If one handles address-space allocation and mapping to physical
addresses in software then one is not limited to 4k, 16k, 4M etc ranges.

As for the time taken there should be relatively few table walks
compared with the number of memory accesses.

Software can arrange to access a reverse page table in any way it wants,
including something with a hashing primary stage.

At least at this point I'd rather see the CPU have instructions which
can help make the reverse lookup faster than it require a page table for
every process.

Overall, it's an idea worth considering. YMMV but as you may gather,
what I do take issue with you on is your blanket comment that there are
no benefits...!

--
James Harris

Dan Cross

2023-12-14 20:41:32 UTC

Post by Dan Cross
It was never clear to me
how a hypervisor could, in general, know the format of the guest
page tables. I know the Disco folks had to make some changes to
Irix to get it to work.

I get it from a hardware perspective: fewer transistors with a
software-managed TLB, but man...so many drawbacks.

Handling a software-managed TLB may be more work, in a sense, but it
gives an OS developer more control, more feedback, more freedom, and
perhaps better opportunities for performance gains - as long as the TLB
is large enough.
Having the hardware carry out a walk of page tables (the only option if
the TLB can is updated by hardware) has long seemed to me like a bad
idea, and it doesn't scale very well as addresses get wider.

On the contrary!
Brendan (BGB) has already covered hardware issues and I respect his
views and experience on that.

I plonked him a long time ago because he seems like a crank.
Has he ever worked on a real, non-hobbyist system?

I trust Waterman a lot more. He chose hardware page tables for
RISC-V. There's a reason for that (see the RISC-V reader).

Post by James Harris
The /software/ problem with forward page
tables is that while they are OK (but still unnecessary) for small
numbers of processes and concomitant address spaces they don't work so
well for larger systems.

Evidence suggests otherwise.

Evidence shows that it scales, and scales pretty darn well.

A soft TLB simply punts the problem to software; software still
has to maintain a mapping of virtual to physical addresses for
each process. While it is true that it need not use an
architecturally-defined format for that mapping, it must still
maintain _some_ mapping that is, by definition, proportional to
the size of the process's memory footprint.

Post by James Harris
By contrast, with reverse page tables you need just ONE table per
machine, no matter how many processes you want to support. Instead of
taking 1/1000th of the address space for every process one would use,
say, 1/500th of the RAM for all processes combined.

First, inverted page tables are not the same thing as software-
managed TLBs; the two may be used in concert, but one does not
automatically imply the other. Second, on very large memory
systems, inverted page tables are wasteful since there's an
entry for each frame of physical memory (and it's conceivable
that not all of physical memory is present!). Third, a single
inverted page table as a global system resource can become a
point of contention in workloads where a process's virtual
address space is being modified frequently. Fourth, fast access
based on a virtual address still requires some sort of lookup
structure that is per-process (since a virtual address space is
presumably a per-process feature of a system).

Post by James Harris
I acknowledge that each process will need forward mappings but can
arrange to reserve them in runs of (first, last) addresses for much
greater space efficiency than you'd get with forward page tables.

Not really.

If one handles address-space allocation and mapping to physical
addresses in software then one is not limited to 4k, 16k, 4M etc ranges.

Eh? that's a function of the MMU, even in systems with soft
TLBs.

Post by James Harris
As for the time taken there should be relatively few table walks
compared with the number of memory accesses.

What makes you think that?

Post by James Harris
Software can arrange to access a reverse page table in any way it wants,
including something with a hashing primary stage.
At least at this point I'd rather see the CPU have instructions which
can help make the reverse lookup faster than it require a page table for
every process.
Overall, it's an idea worth considering. YMMV but as you may gather,
what I do take issue with you on is your blanket comment that there are
no benefits...!

You're conflating software-managed TLBs with inverted page
tables, which is already a mistake. Otherwise, I don't see any
evidence to back up your claim, but I see a lot for the inverse.

- Dan C.

BGB

2023-12-15 04:38:28 UTC

It was never clear to me
how a hypervisor could, in general, know the format of the guest
page tables. I know the Disco folks had to make some changes to
Irix to get it to work.

I get it from a hardware perspective: fewer transistors with a
software-managed TLB, but man...so many drawbacks.

Handling a software-managed TLB may be more work, in a sense, but it
gives an OS developer more control, more feedback, more freedom, and
perhaps better opportunities for performance gains - as long as the TLB
is large enough.
Having the hardware carry out a walk of page tables (the only option if
the TLB can is updated by hardware) has long seemed to me like a bad
idea, and it doesn't scale very well as addresses get wider.

It depends...

But, thanks anyways for the confidence here.

Post by James Harris
Forward page tables take up a significant amount of RAM - as a rule of
thumb let's say up to 1/1000th of the address space, e.g. up to 4M for a
4G address space on x32. While that's no problem for a small number of
processes and the impact can be reduced in many cases it's still
unavoidably true that each time you add a new process you have to add a
new set of page tables. Have 5,000 processes and you need 5,000 sets of
forward page tables. The whole idea is, to use a technical term,
bonkers. As I said, the approach does not scale.
By contrast, with reverse page tables you need just ONE table per
machine, no matter how many processes you want to support. Instead of
taking 1/1000th of the address space for every process one would use,
say, 1/500th of the RAM for all processes combined.

Something like an inverted page-table is more sensible to be used more
as a cache for some other page-table-like structure, not really as a
full replacement for it.

One limiting factor is associativity. Say, for example, if the inverted
page-table is 8-way associative, then one may still have frequent
"misses" if more than 8 commonly used pages happen to land on the same
location in the table (depending on address space layouts, this can
potentially still be a serious issue for performance).

One may or may not have an inverted page table in hardware, depending
mostly on cost tradeoffs:
Like with a page-table walker, it does have the drawback of needing to
have a mechanism to access memory;
But, it can reduce performance overheads by reducing the number of TLB
misses that need to be handled by software.

But, as for page-table alternatives, I had "sorta OK" results with
B-Trees, which can be more space-efficient, however, they do have the
drawback of being significantly slower.

The B-Tree variants mostly make sense for "very large" (and very sparse)
address spaces, whereas for an address space that is 48 bits or less,
page tables work well enough.

A semi-practical alternative is hybrid B-Trees, where the last level is
a page-table, but all of the upper levels are B-Tree.

These can get much of the same advantages for large sparse address
spaces, with a less of a performance impact than a pure B-Tree.

It is also possible to use a hash-table to cache lookups from the B-Tree
part of the table.

For a 48-bit address space with 16K pages, I am admittedly mostly just
using 3 level conventional page-tables.

I had considered B-Trees mostly for an extended 96-bit address space;
mostly because the upper 5 levels of page-table would be "extremely
sparse" (and a hybrid B-Tree could generally collapse everything back
down to 3 levels).

This can happen in some scenarios, but sadly not really in cases where
one is using a pagefile (where page numbers may end up being assigned
"basically at random").

If one handles address-space allocation and mapping to physical
addresses in software then one is not limited to 4k, 16k, 4M etc ranges.

Technically, yes.

You can represent a logical page-size that is any desired multiple of
the page-size used by the TLB.

The main tradeoff then is mostly the relation between page-size and TLB
capacity.

Smaller page size: More TLB misses; Bigger pages: fewer TLB misses.

In my testing, 16K seemed like the local optimum.

For a set-associative TLB design, generally only one page size can be
used at a time. It is possible to use multiple page-sizes with a
fully-associative TLB, but getting to any non-trivial size with a
fully-associative design becomes rather expensive (so, 4 or 8 way
associativity seems like a reasonable limit).

At least on Xilinx FPGAs, there are magic numbers for array sizes:
16 entries: OK (LUTRAM)
FPGA uses half of a 32-entry LUTRAM.
32 entries: Cheapest case (LUTRAM);
64 entries: Also a good value (with LUTRAM);
128 entries: Bad;
Here, it can either use multiple LUTs per bit (LUTRAM),
or use half of a BRAM.
256 entries: OK.
512 entries: Seems to be best case for BRAM.

And, as for TLB size (4-way):
16/32 x4 entry: Bad, very high TLB miss rate.
64 x4 entry: Works, but still a bit small, high miss rate.
TLB miss rate is still fairly high.
128 x4: Just No (bad for resource cost).
256 x4: Good.
512 x4: Also good.

512 x4: With 16K pages, covers roughly 32MB of working-set.

I hadn't much reason to go much bigger than this, as most of the
programs I am messing with have smaller working sets than this.

If putting a TLB in the L1 cache, it can be smaller than this (here, a
32-entry 1-way TLB works well; caching lookups from the main TLB).

Tradeoffs would likely differ for an ASIC, which has comparably cheaper
logic, but SRAM is comparably more expensive.

For an inverted page table, would likely make sense to go a fair bit
larger (say, for example, 4096x8 would cover around 512MB of working set).

There is also a potential tradeoff of whether to XOR the ASID bits
against the table-index. This can reduce miss rates (for multiple
address spaces with overlapping addresses) but hinders the ability to
have "Global Pages".

But, global pages are a bit hit or miss anyways (the use of global pages
can be a foot-gun if one wants to emulate multi-level address
translation for an emulator; since they could potentially "bleed over"
into the emulated environment).

One workaround was to organize ASIDs into groups, with global pages
unable to be shared between groups. Then a range of ASIDs can be
designated as "global pages are forbidden here" (and can be used for
emulator address spaces or similar).

Where, say, global pages can at least have the merit of reducing the
number of TLB misses on process context switches (for any code/data
shared between processes).

Though, one other possibility here could be to split the TLB into two
sub-TLBs:
One with exclusively global pages, with a modulo index;
One with exclusively private pages, with an index XOR'ed with the ASID.

...

Post by James Harris
As for the time taken there should be relatively few table walks
compared with the number of memory accesses.

Yes.

In my testing, the miss rate seems to be generally around 50 to 250 TLB
misses per second (with a 256x4 TLB) on a 50MHz CPU.

Here, that it takes typically around 1000 cycles to service a TLB miss,
doesn't really matter that much.

Comparably more time ends up being spent in the 1kHz timer interrupt
(which has a similar amount of interrupt handling overhead, but happens
1000 times per second).

Though, as a disclaimer, this is with a system where currently all of
the processes are mapped into a shared virtual address space; so,
possibly may not work as well on a system where every process has its
own independent address space. In this case, the idea was to use ACL
checking rather than separate address spaces for memory protection (*1).

Though, is likely to work at least slightly better provided processes
have their memory located at different addresses, such that a context
switch doesn't immediately need to knock the other process's pages out
of the TLB (say, if ASLR is used by default).

A situation where one has a bunch of processes which all use the same
address ranges (such as might happen if one assigns memory addresses
linearly) is basically the worst case in terms of knocking each others'
pages out of the TLB on context switches (so, would likely result in
poor performance).

Though, if "scaling up", implementing hardware support for an inverted
page table or similar may make sense (say, if using multiple address
spaces results in a significant increase in TLB miss rate).

*1: With a virtual memory map sort of like:
0000_xxxxxxxx: Low 4GB, mostly OS use;
0001_xxxxxxxx..3FFF_xxxxxxxx: Shared/Global address space (32TB);
4000_xxxxxxxx..7FFF_xxxxxxxx: Local/Private address space (64TB);
8000_xxxxxxxx..BFFF_xxxxxxxx: OS/Kernel address space (32TB);
C000_xxxxxxxx..FFFF_xxxxxxxx: Physical addresses and MMIO.

Local/Private space will serve a similar role to traditional separate
address spaces (things like process heap memory and similar go here).
Addresses into this space will not be valid from one process to another
(with addresses being assigned using a random number generator).

Things like executable code and "GlobalAlloc" memory, etc, go into the
global address range (but may still be subject to ACL checking).

In this case, the virtual address space being 48 bits, with the high
16-bits of the pointer mostly left for "other stuff" (mostly type-tags
and similar).

If anything, a more immediate priority would be to add register banks or
similar such that interrupt handlers can be faster.

Though, in turn, most of the overhead with interrupt handling is due to
the L1 cache misses; since pretty much any time an interrupt happens,
the running program has long since knocked all of the relevant cache
lines out of the L1 cache.

Post by James Harris
Software can arrange to access a reverse page table in any way it wants,
including something with a hashing primary stage.

Possibly.

Post by James Harris
At least at this point I'd rather see the CPU have instructions which
can help make the reverse lookup faster than it require a page table for
every process.
Overall, it's an idea worth considering. YMMV but as you may gather,
what I do take issue with you on is your blanket comment that there are
no benefits...!

Agreed.

One can do a few things:
Use layouts other than a nested page table;
Implement custom or non-standard memory protection schemes;
Implement more advanced features (such as nested translation) without
needing special hardware support;
...

For example, last I looked, no one is exactly jumping on things like
adding file-system-like ACL checks onto a convention hardware
page-walking MMU (they will give you "protection rings" or similar and
call it done).

Also works pretty OK for being the cheapest option for a soft-processor
implemented on an FPGA.

Dan Cross

2023-12-13 15:29:47 UTC

Post by Dan Cross
It was never clear to me
how a hypervisor could, in general, know the format of the guest
page tables. I know the Disco folks had to make some changes to
Irix to get it to work.

I get it from a hardware perspective: fewer transistors with a
software-managed TLB, but man...so many drawbacks.

Citation needed.

Post by James Harris
Having the hardware carry out a walk of page tables (the only option if
the TLB can is updated by hardware) has long seemed to me like a bad
idea, and it doesn't scale very well as addresses get wider.

This seems like a claim that isn't going to stand up to close
scrutiny, let alone evidence.

- Dan C.

James Harris

2023-12-14 20:16:59 UTC

Post by Dan Cross
It was never clear to me
how a hypervisor could, in general, know the format of the guest
page tables. I know the Disco folks had to make some changes to
Irix to get it to work.

I get it from a hardware perspective: fewer transistors with a
software-managed TLB, but man...so many drawbacks.

Citation needed.

This seems like a claim that isn't going to stand up to close
scrutiny, let alone evidence.

Is any of that not covered in the reply I've just made to Scott?

--
James Harris

Dan Cross

2023-12-14 20:42:24 UTC

Post by Dan Cross
It was never clear to me
how a hypervisor could, in general, know the format of the guest
page tables. I know the Disco folks had to make some changes to
Irix to get it to work.

I get it from a hardware perspective: fewer transistors with a
software-managed TLB, but man...so many drawbacks.

Citation needed.

This seems like a claim that isn't going to stand up to close
scrutiny, let alone evidence.

Is any of that not covered in the reply I've just made to Scott?

Well, you posted no citations and no real evidence though you
did make a lot of claims.

- Dan C.

Scott Lurndal

2023-12-14 22:23:03 UTC

Post by Dan Cross
It was never clear to me
how a hypervisor could, in general, know the format of the guest
page tables. I know the Disco folks had to make some changes to
Irix to get it to work.

I get it from a hardware perspective: fewer transistors with a
software-managed TLB, but man...so many drawbacks.

Citation needed.

This seems like a claim that isn't going to stand up to close
scrutiny, let alone evidence.

Is any of that not covered in the reply I've just made to Scott?

Well, you posted no citations and no real evidence though you
did make a lot of claims.

None of which were particularly compelling. Perhaps his ideas
would be effective on very small microcontrollers, but not on
any large multi-user or multi-tenant server CPU.

My development team's server is currently running 740 tasks
on a 64 processor machine (64GB memory). Scales just fine.

Dan Cross

2023-12-15 12:26:53 UTC

[snip]
None of which were particularly compelling. Perhaps his ideas
would be effective on very small microcontrollers, but not on
any large multi-user or multi-tenant server CPU.
My development team's server is currently running 740 tasks
on a 64 processor machine (64GB memory). Scales just fine.

Indeed. Large x86 machines are pushing a thousand cores
and TiB of RAM using architecturally defined page tables.
To claim it "doesn't scale" is ludicrous.

- Dan C.

muta...@gmail.com

2023-03-22 22:34:31 UTC

Post by wolfgang kern
me too like to see attempts to create an UEFI compliant OS from scratch
w/o reusing C-libraries nor copies from already existing Lonnix/M$.

What's wrong with using a C library to do the UEFI calls?
easy answer: it is C, aka limited+bloated+weird HLL-stuff.

Like UEFI itself?

The C library that I use that wraps the UEFI calls is 46k in total
(executable).

I suspect the requirements for UEFI are a modern machine
with much more than 46k of RAM.

You could use the generated assembler, and you could trim
it down, but is there a purpose to that?

You mentioned you want to see "attempts" (multiple, and
presumably in assembler) - why? Are you going to use them,
or you just want to look at them, even if they all have the
same design?

BFN. Paul.

wolfgang kern

2023-03-23 08:39:04 UTC

Post by wolfgang kern
me too like to see attempts to create an UEFI compliant OS from scratch
w/o reusing C-libraries nor copies from already existing Lonnix/M$.

What's wrong with using a C library to do the UEFI calls?
easy answer: it is C, aka limited+bloated+weird HLL-stuff.

Like UEFI itself?

yes.

Post by ***@gmail.com
The C library that I use that wraps the UEFI calls is 46k in total
(executable).
I suspect the requirements for UEFI are a modern machine
with much more than 46k of RAM.

it's unfortunately the other way around:
I can't buy any new PC-hardware w/o UEFI on it.
an alternative option would be Raspberry Pi II :)

Post by ***@gmail.com
You could use the generated assembler, and you could trim
it down, but is there a purpose to that?

there's is no way to trim C-created code. Concept of C is wrong.

my OS need the BIOS only to load the first 1024 byte, it then deploys
itself with everything required incl. Debugger, GUI and device drivers.
So my Function-Call link is less than 46k namely just 2 byte:

mov eax,...;all parameters are in registers either VAL or PTR.
int 0x7f ;works in all CPU modes except BigReal(RM16,32bit CS).
...........;returned parameters are in registers by VAL or PTR

Post by ***@gmail.com
You mentioned you want to see "attempts" (multiple, and
presumably in assembler) - why? Are you going to use them,
or you just want to look at them, even if they all have the
same design?

I'm interested how and where to start such an attempt, but I wouldn't
"use" it because my program style was/is uncommon anyway.
for now I feel too old for a new start, but if all stars align ...
__
wolfgang

muta...@gmail.com

2023-03-23 09:08:58 UTC

Post by ***@gmail.com
The C library that I use that wraps the UEFI calls is 46k in total
(executable).
I suspect the requirements for UEFI are a modern machine
with much more than 46k of RAM.

I can't buy any new PC-hardware w/o UEFI on it.

I don't understand.

What's wrong with UEFI?

What's wrong with 46k as part of a BOOTX64.EFI?

Post by ***@gmail.com
You could use the generated assembler, and you could trim
it down, but is there a purpose to that?

there's is no way to trim C-created code. Concept of C is wrong.

What I meant is that the 46k above can be trimmed
down. You don't need the entire C library linked in,
depending on what you want to do.

I'm interested how and where to start such an attempt,

You want to know how to write a hello world using UEFI?

There are examples on the net, and I have one too
(also on the net - but you have to go back to an older
version - also on the net - of the code to see that,
because I have since moved on from "hello world".

But the "hello world" code is still there, just commented out:

https://sourceforge.net/p/pdos/gitcode/ci/master/tree/pdpclib/efistart.c

if ((Status = print_string ("Hello, world!\n")))

static EFI_STATUS print_string (char *str) {

BFN. Paul.

muta...@gmail.com

2023-03-23 09:12:33 UTC

Post by ***@gmail.com
if ((Status = print_string ("Hello, world!\n")))

Clarification - that code doesn't use a C library,
even though it is written in C. It simply calls
the UEFI interface.

And I can provide the generated x64 code if you
want to see that.

So that would be standalone, small assembler code
to print hello world.

BFN. Paul.

wolfgang kern

2023-03-23 11:25:21 UTC

Post by ***@gmail.com
The C library that I use that wraps the UEFI calls is 46k in total
(executable).
I suspect the requirements for UEFI are a modern machine
with much more than 46k of RAM.

I can't buy any new PC-hardware w/o UEFI on it.

I don't understand.
What's wrong with UEFI?
What's wrong with 46k as part of a BOOTX64.EFI?

first it's HLL-styled stack abusing bloatware. And I cannot boot my OS
on it nor use my own tools. it needs on INT13 to start.
So what is the UEFI equivalent to INT13 and how can I make it boot my
old stuff from either CD or USB ?

Post by wolfgang kern
I'm interested how and where to start such an attempt,

You want to know how to write a hello world using UEFI?

No! I like to see what tools are required to format and prepare a new PC
for my needs.
__
wolfgang

muta...@gmail.com

2023-03-23 13:15:56 UTC

Post by ***@gmail.com
The C library that I use that wraps the UEFI calls is 46k in total
(executable).
I suspect the requirements for UEFI are a modern machine
with much more than 46k of RAM.

I can't buy any new PC-hardware w/o UEFI on it.

I don't understand.
What's wrong with UEFI?
What's wrong with 46k as part of a BOOTX64.EFI?

Post by wolfgang kern
I'm interested how and where to start such an attempt,

You want to know how to write a hello world using UEFI?

No! I like to see what tools are required to format and prepare a new PC
for my needs.
__
wolfgang

Not on my computer at the moment so can't reply properly.

You said you just needed your bootloader loaded and then you no longer use the bios, right?

In that case, the equivalent of int 13h is nothing.

You just need to name your bootloader bootx64.efi

Uefi has the bloat in firmware to recognize an entire file system.

There are some other issues though.

muta...@gmail.com

2023-03-23 14:45:56 UTC

Ok, finally back at my computer.

Post by wolfgang kern
So what is the UEFI equivalent to INT13 and how can I make it boot my
old stuff from either CD or USB ?

As mentioned, if all you need is a bit of code loaded,
you don't need to anything at all except plonk a
bootx64.efi into \efi\boot of a FAT-formatted disk.

Here is some assembler code:

C:\devel\pdos\pdpclib\xxx3>type efi64.asm
# Written by Paul Edwards
# Released to the public domain

.code64
.intel_syntax noprefix

.globl efimain
efimain:

# ret is 1 byte, jmp is 2 bytes
# So for testing, comment out the jmp and add 2 rets

# xxx: jmp xxx
ret
ret

ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret
ret

C:\devel\pdos\pdpclib\xxx3>

The assembler can be compiled and linked like this
(with mingw64 for Windows)

C:\devel\pdos\pdpclib\xxx3>type doit.bat
x86_64-w64-mingw32-as -o efi64.o efi64.asm
x86_64-w64-mingw32-ld -s -subsystem 10 -e efimain -nostdlib -o bootx64.efi efi64.o

C:\devel\pdos\pdpclib\xxx3>

Here are two executables:

C:\vbox\x64test\efi\boot>dir
Volume in drive C has no label.
Volume Serial Number is 4E58-AF11

Directory of C:\vbox\x64test\efi\boot

2023-03-23 22:17 <DIR> .
2023-03-23 22:17 <DIR> ..
2023-03-23 22:17 2,048 bootx64.efi
2023-03-23 22:15 2,048 loop.efi

C:\vbox\x64test\efi\boot>hexdump loop.efi
000000 4D5A9000 03000000 04000000 FFFF0000 MZ..............
000010 B8000000 00000000 40000000 00000000 ***@.......
000020 00000000 00000000 00000000 00000000 ................
000030 00000000 00000000 00000000 80000000 ................
000040 0E1FBA0E 00B409CD 21B8014C CD215468 ........!..L.!Th
000050 69732070 726F6772 616D2063 616E6E6F is program canno
000060 74206265 2072756E 20696E20 444F5320 t be run in DOS
000070 6D6F6465 2E0D0D0A 24000000 00000000 mode....$.......
000080 50450000 64860200 F25E1C64 00000000 PE..d....^.d....
000090 00000000 F0002E02 0B020224 00020000 ...........$....
0000A0 00020000 00000000 00100000 00100000 ................
0000B0 00000040 01000000 00100000 00020000 ***@............
0000C0 04000000 00000000 05000200 00000000 ................
0000D0 00300000 00040000 79FD0000 0A006001 .0......y.....`.
0000E0 00002000 00000000 00100000 00000000 .. .............
0000F0 00001000 00000000 00100000 00000000 ................
000100 00000000 10000000 00000000 00000000 ................
000110 00200000 14000000 00000000 00000000 . ..............
000120 00000000 00000000 00000000 00000000 ................
000130 00000000 00000000 00000000 00000000 ................
000140 00000000 00000000 00000000 00000000 ................
000150 00000000 00000000 00000000 00000000 ................
000160 00000000 00000000 00000000 00000000 ................
000170 00000000 00000000 00000000 00000000 ................
000180 00000000 00000000 2E746578 74000000 .........text...
000190 60000000 00100000 00020000 00040000 `...............
0001A0 00000000 00000000 00000000 20005060 ............ .P`
0001B0 2E696461 74610000 14000000 00200000 .idata....... ..
0001C0 00020000 00060000 00000000 00000000 ................
0001D0 00000000 400030C0 00000000 00000000 ***@.0.........
0001E0 00000000 00000000 00000000 00000000 ................
0001F0 00000000 00000000 00000000 00000000 ................
000200 00000000 00000000 00000000 00000000 ................
000210 00000000 00000000 00000000 00000000 ................
000220 00000000 00000000 00000000 00000000 ................
000230 00000000 00000000 00000000 00000000 ................
000240 00000000 00000000 00000000 00000000 ................
000250 00000000 00000000 00000000 00000000 ................
000260 00000000 00000000 00000000 00000000 ................
000270 00000000 00000000 00000000 00000000 ................
000280 00000000 00000000 00000000 00000000 ................
000290 00000000 00000000 00000000 00000000 ................
0002A0 00000000 00000000 00000000 00000000 ................
0002B0 00000000 00000000 00000000 00000000 ................
0002C0 00000000 00000000 00000000 00000000 ................
0002D0 00000000 00000000 00000000 00000000 ................
0002E0 00000000 00000000 00000000 00000000 ................
0002F0 00000000 00000000 00000000 00000000 ................
000300 00000000 00000000 00000000 00000000 ................
000310 00000000 00000000 00000000 00000000 ................
000320 00000000 00000000 00000000 00000000 ................
000330 00000000 00000000 00000000 00000000 ................
000340 00000000 00000000 00000000 00000000 ................
000350 00000000 00000000 00000000 00000000 ................
000360 00000000 00000000 00000000 00000000 ................
000370 00000000 00000000 00000000 00000000 ................
000380 00000000 00000000 00000000 00000000 ................
000390 00000000 00000000 00000000 00000000 ................
0003A0 00000000 00000000 00000000 00000000 ................
0003B0 00000000 00000000 00000000 00000000 ................
0003C0 00000000 00000000 00000000 00000000 ................
0003D0 00000000 00000000 00000000 00000000 ................
0003E0 00000000 00000000 00000000 00000000 ................
0003F0 00000000 00000000 00000000 00000000 ................
000400 EBFEC3C3 C3C3C3C3 C3C3C3C3 C3C3C3C3 ................
000410 C3C3C3C3 C3C3C3C3 C3C3C3C3 C3C3C3C3 ................
000420 C3C3C3C3 C3C3C3C3 C3C3C3C3 C3C3C3C3 ................
000430 C3C3C3C3 C3C3C3C3 C3C3C3C3 C3C39090 ................
000440 FFFFFFFF FFFFFFFF 00000000 00000000 ................
000450 FFFFFFFF FFFFFFFF 00000000 00000000 ................
000460 00000000 00000000 00000000 00000000 ................
000470 00000000 00000000 00000000 00000000 ................
000480 00000000 00000000 00000000 00000000 ................
000490 00000000 00000000 00000000 00000000 ................
0004A0 00000000 00000000 00000000 00000000 ................
0004B0 00000000 00000000 00000000 00000000 ................
0004C0 00000000 00000000 00000000 00000000 ................
0004D0 00000000 00000000 00000000 00000000 ................
0004E0 00000000 00000000 00000000 00000000 ................
0004F0 00000000 00000000 00000000 00000000 ................
000500 00000000 00000000 00000000 00000000 ................
000510 00000000 00000000 00000000 00000000 ................
000520 00000000 00000000 00000000 00000000 ................
000530 00000000 00000000 00000000 00000000 ................
000540 00000000 00000000 00000000 00000000 ................
000550 00000000 00000000 00000000 00000000 ................
000560 00000000 00000000 00000000 00000000 ................
000570 00000000 00000000 00000000 00000000 ................
000580 00000000 00000000 00000000 00000000 ................
000590 00000000 00000000 00000000 00000000 ................
0005A0 00000000 00000000 00000000 00000000 ................
0005B0 00000000 00000000 00000000 00000000 ................
0005C0 00000000 00000000 00000000 00000000 ................
0005D0 00000000 00000000 00000000 00000000 ................
0005E0 00000000 00000000 00000000 00000000 ................
0005F0 00000000 00000000 00000000 00000000 ................
000600 00000000 00000000 00000000 00000000 ................
000610 00000000 00000000 00000000 00000000 ................
000620 00000000 00000000 00000000 00000000 ................
000630 00000000 00000000 00000000 00000000 ................
000640 00000000 00000000 00000000 00000000 ................
000650 00000000 00000000 00000000 00000000 ................
000660 00000000 00000000 00000000 00000000 ................
000670 00000000 00000000 00000000 00000000 ................
000680 00000000 00000000 00000000 00000000 ................
000690 00000000 00000000 00000000 00000000 ................
0006A0 00000000 00000000 00000000 00000000 ................
0006B0 00000000 00000000 00000000 00000000 ................
0006C0 00000000 00000000 00000000 00000000 ................
0006D0 00000000 00000000 00000000 00000000 ................
0006E0 00000000 00000000 00000000 00000000 ................
0006F0 00000000 00000000 00000000 00000000 ................
000700 00000000 00000000 00000000 00000000 ................
000710 00000000 00000000 00000000 00000000 ................
000720 00000000 00000000 00000000 00000000 ................
000730 00000000 00000000 00000000 00000000 ................
000740 00000000 00000000 00000000 00000000 ................
000750 00000000 00000000 00000000 00000000 ................
000760 00000000 00000000 00000000 00000000 ................
000770 00000000 00000000 00000000 00000000 ................
000780 00000000 00000000 00000000 00000000 ................
000790 00000000 00000000 00000000 00000000 ................
0007A0 00000000 00000000 00000000 00000000 ................
0007B0 00000000 00000000 00000000 00000000 ................
0007C0 00000000 00000000 00000000 00000000 ................
0007D0 00000000 00000000 00000000 00000000 ................
0007E0 00000000 00000000 00000000 00000000 ................
0007F0 00000000 00000000 00000000 00000000 ................

C:\vbox\x64test\efi\boot>

C:\vbox\x64test\efi\boot>hexdump bootx64.efi
000000 4D5A9000 03000000 04000000 FFFF0000 MZ..............
000010 B8000000 00000000 40000000 00000000 ***@.......
000020 00000000 00000000 00000000 00000000 ................
000030 00000000 00000000 00000000 80000000 ................
000040 0E1FBA0E 00B409CD 21B8014C CD215468 ........!..L.!Th
000050 69732070 726F6772 616D2063 616E6E6F is program canno
000060 74206265 2072756E 20696E20 444F5320 t be run in DOS
000070 6D6F6465 2E0D0D0A 24000000 00000000 mode....$.......
000080 50450000 64860200 825F1C64 00000000 PE..d...._.d....
000090 00000000 F0002E02 0B020224 00020000 ...........$....
0000A0 00020000 00000000 00100000 00100000 ................
0000B0 00000040 01000000 00100000 00020000 ***@............
0000C0 04000000 00000000 05000200 00000000 ................
0000D0 00300000 00040000 E1C20000 0A006001 .0............`.
0000E0 00002000 00000000 00100000 00000000 .. .............
0000F0 00001000 00000000 00100000 00000000 ................
000100 00000000 10000000 00000000 00000000 ................
000110 00200000 14000000 00000000 00000000 . ..............
000120 00000000 00000000 00000000 00000000 ................
000130 00000000 00000000 00000000 00000000 ................
000140 00000000 00000000 00000000 00000000 ................
000150 00000000 00000000 00000000 00000000 ................
000160 00000000 00000000 00000000 00000000 ................
000170 00000000 00000000 00000000 00000000 ................
000180 00000000 00000000 2E746578 74000000 .........text...
000190 60000000 00100000 00020000 00040000 `...............
0001A0 00000000 00000000 00000000 20005060 ............ .P`
0001B0 2E696461 74610000 14000000 00200000 .idata....... ..
0001C0 00020000 00060000 00000000 00000000 ................
0001D0 00000000 400030C0 00000000 00000000 ***@.0.........
0001E0 00000000 00000000 00000000 00000000 ................
0001F0 00000000 00000000 00000000 00000000 ................
000200 00000000 00000000 00000000 00000000 ................
000210 00000000 00000000 00000000 00000000 ................
000220 00000000 00000000 00000000 00000000 ................
000230 00000000 00000000 00000000 00000000 ................
000240 00000000 00000000 00000000 00000000 ................
000250 00000000 00000000 00000000 00000000 ................
000260 00000000 00000000 00000000 00000000 ................
000270 00000000 00000000 00000000 00000000 ................
000280 00000000 00000000 00000000 00000000 ................
000290 00000000 00000000 00000000 00000000 ................
0002A0 00000000 00000000 00000000 00000000 ................
0002B0 00000000 00000000 00000000 00000000 ................
0002C0 00000000 00000000 00000000 00000000 ................
0002D0 00000000 00000000 00000000 00000000 ................
0002E0 00000000 00000000 00000000 00000000 ................
0002F0 00000000 00000000 00000000 00000000 ................
000300 00000000 00000000 00000000 00000000 ................
000310 00000000 00000000 00000000 00000000 ................
000320 00000000 00000000 00000000 00000000 ................
000330 00000000 00000000 00000000 00000000 ................
000340 00000000 00000000 00000000 00000000 ................
000350 00000000 00000000 00000000 00000000 ................
000360 00000000 00000000 00000000 00000000 ................
000370 00000000 00000000 00000000 00000000 ................
000380 00000000 00000000 00000000 00000000 ................
000390 00000000 00000000 00000000 00000000 ................
0003A0 00000000 00000000 00000000 00000000 ................
0003B0 00000000 00000000 00000000 00000000 ................
0003C0 00000000 00000000 00000000 00000000 ................
0003D0 00000000 00000000 00000000 00000000 ................
0003E0 00000000 00000000 00000000 00000000 ................
0003F0 00000000 00000000 00000000 00000000 ................
000400 C3C3C3C3 C3C3C3C3 C3C3C3C3 C3C3C3C3 ................
000410 C3C3C3C3 C3C3C3C3 C3C3C3C3 C3C3C3C3 ................
000420 C3C3C3C3 C3C3C3C3 C3C3C3C3 C3C3C3C3 ................
000430 C3C3C3C3 C3C3C3C3 C3C3C3C3 C3C39090 ................
000440 FFFFFFFF FFFFFFFF 00000000 00000000 ................
000450 FFFFFFFF FFFFFFFF 00000000 00000000 ................
000460 00000000 00000000 00000000 00000000 ................
000470 00000000 00000000 00000000 00000000 ................
000480 00000000 00000000 00000000 00000000 ................
000490 00000000 00000000 00000000 00000000 ................
0004A0 00000000 00000000 00000000 00000000 ................
0004B0 00000000 00000000 00000000 00000000 ................
0004C0 00000000 00000000 00000000 00000000 ................
0004D0 00000000 00000000 00000000 00000000 ................
0004E0 00000000 00000000 00000000 00000000 ................
0004F0 00000000 00000000 00000000 00000000 ................
000500 00000000 00000000 00000000 00000000 ................
000510 00000000 00000000 00000000 00000000 ................
000520 00000000 00000000 00000000 00000000 ................
000530 00000000 00000000 00000000 00000000 ................
000540 00000000 00000000 00000000 00000000 ................
000550 00000000 00000000 00000000 00000000 ................
000560 00000000 00000000 00000000 00000000 ................
000570 00000000 00000000 00000000 00000000 ................
000580 00000000 00000000 00000000 00000000 ................
000590 00000000 00000000 00000000 00000000 ................
0005A0 00000000 00000000 00000000 00000000 ................
0005B0 00000000 00000000 00000000 00000000 ................
0005C0 00000000 00000000 00000000 00000000 ................
0005D0 00000000 00000000 00000000 00000000 ................
0005E0 00000000 00000000 00000000 00000000 ................
0005F0 00000000 00000000 00000000 00000000 ................
000600 00000000 00000000 00000000 00000000 ................
000610 00000000 00000000 00000000 00000000 ................
000620 00000000 00000000 00000000 00000000 ................
000630 00000000 00000000 00000000 00000000 ................
000640 00000000 00000000 00000000 00000000 ................
000650 00000000 00000000 00000000 00000000 ................
000660 00000000 00000000 00000000 00000000 ................
000670 00000000 00000000 00000000 00000000 ................
000680 00000000 00000000 00000000 00000000 ................
000690 00000000 00000000 00000000 00000000 ................
0006A0 00000000 00000000 00000000 00000000 ................
0006B0 00000000 00000000 00000000 00000000 ................
0006C0 00000000 00000000 00000000 00000000 ................
0006D0 00000000 00000000 00000000 00000000 ................
0006E0 00000000 00000000 00000000 00000000 ................
0006F0 00000000 00000000 00000000 00000000 ................
000700 00000000 00000000 00000000 00000000 ................
000710 00000000 00000000 00000000 00000000 ................
000720 00000000 00000000 00000000 00000000 ................
000730 00000000 00000000 00000000 00000000 ................
000740 00000000 00000000 00000000 00000000 ................
000750 00000000 00000000 00000000 00000000 ................
000760 00000000 00000000 00000000 00000000 ................
000770 00000000 00000000 00000000 00000000 ................
000780 00000000 00000000 00000000 00000000 ................
000790 00000000 00000000 00000000 00000000 ................
0007A0 00000000 00000000 00000000 00000000 ................
0007B0 00000000 00000000 00000000 00000000 ................
0007C0 00000000 00000000 00000000 00000000 ................
0007D0 00000000 00000000 00000000 00000000 ................
0007E0 00000000 00000000 00000000 00000000 ................
0007F0 00000000 00000000 00000000 00000000 ................

C:\vbox\x64test\efi\boot>

If you rename loop.efi to bootx64.efi, you will get an
infinite loop as the first instruction.

bootx64.efi as it stands just returns immediately (via
ret) so you will return to your BIOS or a shell may be
executed or something else.

I included a whole lot of "ret" instructions as padding
so that you can zap your own machine code in to do
further testing.

I didn't use "nop" because the assembler pads with nop.

If you zap your own machine code, the required checksum
of the PE executable will change - I don't know if all/any
UEFI actually check that. The location of the checksum
is presumably one of these:

C:\vbox\x64test\efi\boot>fc /b loop.efi bootx64.efi
Comparing files loop.efi and BOOTX64.EFI
00000088: F2 82
00000089: 5E 5F
000000D8: 79 E1
000000D9: FD C2
00000400: EB C3
00000401: FE C3

That's strange - a recompilation changed both:

C:\devel\pdos\pdpclib\xxx3>fc /b \vbox\x64test\efi\boot\bootx64.efi bootx64.efi
Comparing files \VBOX\X64TEST\EFI\BOOT\bootx64.efi and BOOTX64.EFI
00000088: 82 49
00000089: 5F 64
000000D8: E1 A8
000000D9: C2 C7

Maybe the timestamp is included in the checksum.

So take a guess.

You'll presumably need to disable secure boot in the BIOS too.

Issues:

1. I'm not sure you will be in ring 0. Maybe try writing to the serial
port - I don't think you can write to b8000 because I think you
will be in graphics mode.

2. The loop will terminate after a while (minutes?) because of a
watchdog timer. There is a UEFI call to disable the watchdog.
I'm not sure you will need to do that if your bootloader does
something substantial.

3. As far as I know, most people do a UEFI call to "exit boot
services" before they start doing something substantial.
I don't know if you can bypass that.

So you may need to do 0, 1 or 2 UEFI calls.

I can provide the offsets if required:

C:\devel\pdos\pdpclib>grep -i exit efi.h
efi.h: void *Exit;
efi.h: void *ExitBootServices;

C:\devel\pdos\pdpclib>grep -i watchdog efi.h
efi.h: typedef EFI_STATUS (EFIAPI *EFI_SET_WATCHDOG_TIMER) (IN UINTN Timeout, IN UINT64 WatchdogCode, IN UINTN DataSize, IN CHAR16 *WatchdogData);
efi.h: EFI_SET_WATCHDOG_TIMER SetWatchdogTimer;

C:\devel\pdos\pdpclib>

But I've only tested the watchdog.

BFN. Paul.

wolfgang kern

2023-03-23 17:37:36 UTC

Post by wolfgang kern
So what is the UEFI equivalent to INT13 and how can I make it boot my
old stuff from either CD or USB ?

As mentioned, if all you need is a bit of code loaded,
you don't need to anything at all except plonk a
bootx64.efi into \efi\boot of a FAT-formatted disk.

here I have my first problem:
with a new assembled PC I get a main-board with a CD that contains only
a few drivers for windoze loonix and Mac, nothing boot able on it.
the HDs are all empty, not formatted in any usable mode.

So it would need me to buy a win11-DVD which I wont ever do.
any suggestion ? I'd need a boot able formatting tool first.

Next demand would be a tool which can create folders.
And then such a tool must be able to copy my OS-boot-image
into a bootx64.efi perhaps at a certain offset dunno yet how
.efi files were organized (I may find some info on the net)
but the hex-dumps below show that you created M$ PE-files.

one more problem: my OS-images incl. boot-part are just a bunch of
consecutive sectors (can be seen and stored as .bin files)
no header, no redirection, very own checksum algo.
thanks for trying

Post by ***@gmail.com
C:\devel\pdos\pdpclib\xxx3>type efi64.asm
# Written by Paul Edwards
# Released to the public domain
.code64
.intel_syntax noprefix
.globl efimain
# ret is 1 byte, jmp is 2 bytes
# So for testing, comment out the jmp and add 2 rets

dup 64 ret ;wouldn't one be enough?
;where would this return go to ?

so I assume the first 64 bytes are special purpose ?

[...]there is no assembler :) and rare ever will be.

OK I'll keep the trail for reference (2K PE format).
__
wolfgang

Post by ***@gmail.com
C:\vbox\x64test\efi\boot>dir
Volume in drive C has no label.
Volume Serial Number is 4E58-AF11
Directory of C:\vbox\x64test\efi\boot
2023-03-23 22:17 <DIR> .
2023-03-23 22:17 <DIR> ..
2023-03-23 22:17 2,048 bootx64.efi
2023-03-23 22:15 2,048 loop.efi
C:\vbox\x64test\efi\boot>hexdump loop.efi
000000 4D5A9000 03000000 04000000 FFFF0000 MZ..............
000020 00000000 00000000 00000000 00000000 ................
000030 00000000 00000000 00000000 80000000 ................
000040 0E1FBA0E 00B409CD 21B8014C CD215468 ........!..L.!Th
000050 69732070 726F6772 616D2063 616E6E6F is program canno
000060 74206265 2072756E 20696E20 444F5320 t be run in DOS
000070 6D6F6465 2E0D0D0A 24000000 00000000 mode....$.......
000080 50450000 64860200 F25E1C64 00000000 PE..d....^.d....
000090 00000000 F0002E02 0B020224 00020000 ...........$....
0000A0 00020000 00000000 00100000 00100000 ................
0000C0 04000000 00000000 05000200 00000000 ................
0000D0 00300000 00040000 79FD0000 0A006001 .0......y.....`.
0000E0 00002000 00000000 00100000 00000000 .. .............
0000F0 00001000 00000000 00100000 00000000 ................
000100 00000000 10000000 00000000 00000000 ................
000110 00200000 14000000 00000000 00000000 . ..............
000120 00000000 00000000 00000000 00000000 ................
000130 00000000 00000000 00000000 00000000 ................
000140 00000000 00000000 00000000 00000000 ................
000150 00000000 00000000 00000000 00000000 ................
000160 00000000 00000000 00000000 00000000 ................
000170 00000000 00000000 00000000 00000000 ................
000180 00000000 00000000 2E746578 74000000 .........text...
000190 60000000 00100000 00020000 00040000 `...............
0001A0 00000000 00000000 00000000 20005060 ............ .P`
0001B0 2E696461 74610000 14000000 00200000 .idata....... ..
0001C0 00020000 00060000 00000000 00000000 ................
0001E0 00000000 00000000 00000000 00000000 ................
0001F0 00000000 00000000 00000000 00000000 ................
000200 00000000 00000000 00000000 00000000 ................
000210 00000000 00000000 00000000 00000000 ................
000220 00000000 00000000 00000000 00000000 ................
000230 00000000 00000000 00000000 00000000 ................
000240 00000000 00000000 00000000 00000000 ................
000250 00000000 00000000 00000000 00000000 ................
000260 00000000 00000000 00000000 00000000 ................
000270 00000000 00000000 00000000 00000000 ................
000280 00000000 00000000 00000000 00000000 ................
000290 00000000 00000000 00000000 00000000 ................
0002A0 00000000 00000000 00000000 00000000 ................
0002B0 00000000 00000000 00000000 00000000 ................
0002C0 00000000 00000000 00000000 00000000 ................
0002D0 00000000 00000000 00000000 00000000 ................
0002E0 00000000 00000000 00000000 00000000 ................
0002F0 00000000 00000000 00000000 00000000 ................
000300 00000000 00000000 00000000 00000000 ................
000310 00000000 00000000 00000000 00000000 ................
000320 00000000 00000000 00000000 00000000 ................
000330 00000000 00000000 00000000 00000000 ................
000340 00000000 00000000 00000000 00000000 ................
000350 00000000 00000000 00000000 00000000 ................
000360 00000000 00000000 00000000 00000000 ................
000370 00000000 00000000 00000000 00000000 ................
000380 00000000 00000000 00000000 00000000 ................
000390 00000000 00000000 00000000 00000000 ................
0003A0 00000000 00000000 00000000 00000000 ................
0003B0 00000000 00000000 00000000 00000000 ................
0003C0 00000000 00000000 00000000 00000000 ................
0003D0 00000000 00000000 00000000 00000000 ................
0003E0 00000000 00000000 00000000 00000000 ................
0003F0 00000000 00000000 00000000 00000000 ................
000400 EBFEC3C3 C3C3C3C3 C3C3C3C3 C3C3C3C3 ................
000410 C3C3C3C3 C3C3C3C3 C3C3C3C3 C3C3C3C3 ................
000420 C3C3C3C3 C3C3C3C3 C3C3C3C3 C3C3C3C3 ................
000430 C3C3C3C3 C3C3C3C3 C3C3C3C3 C3C39090 ................
000440 FFFFFFFF FFFFFFFF 00000000 00000000 ................
000450 FFFFFFFF FFFFFFFF 00000000 00000000 ................
000460 00000000 00000000 00000000 00000000 ................
000470 00000000 00000000 00000000 00000000 ................
000480 00000000 00000000 00000000 00000000 ................
000490 00000000 00000000 00000000 00000000 ................
0004A0 00000000 00000000 00000000 00000000 ................
0004B0 00000000 00000000 00000000 00000000 ................
0004C0 00000000 00000000 00000000 00000000 ................
0004D0 00000000 00000000 00000000 00000000 ................
0004E0 00000000 00000000 00000000 00000000 ................
0004F0 00000000 00000000 00000000 00000000 ................
000500 00000000 00000000 00000000 00000000 ................
000510 00000000 00000000 00000000 00000000 ................
000520 00000000 00000000 00000000 00000000 ................
000530 00000000 00000000 00000000 00000000 ................
000540 00000000 00000000 00000000 00000000 ................
000550 00000000 00000000 00000000 00000000 ................
000560 00000000 00000000 00000000 00000000 ................
000570 00000000 00000000 00000000 00000000 ................
000580 00000000 00000000 00000000 00000000 ................
000590 00000000 00000000 00000000 00000000 ................
0005A0 00000000 00000000 00000000 00000000 ................
0005B0 00000000 00000000 00000000 00000000 ................
0005C0 00000000 00000000 00000000 00000000 ................
0005D0 00000000 00000000 00000000 00000000 ................
0005E0 00000000 00000000 00000000 00000000 ................
0005F0 00000000 00000000 00000000 00000000 ................
000600 00000000 00000000 00000000 00000000 ................
000610 00000000 00000000 00000000 00000000 ................
000620 00000000 00000000 00000000 00000000 ................
000630 00000000 00000000 00000000 00000000 ................
000640 00000000 00000000 00000000 00000000 ................
000650 00000000 00000000 00000000 00000000 ................
000660 00000000 00000000 00000000 00000000 ................
000670 00000000 00000000 00000000 00000000 ................
000680 00000000 00000000 00000000 00000000 ................
000690 00000000 00000000 00000000 00000000 ................
0006A0 00000000 00000000 00000000 00000000 ................
0006B0 00000000 00000000 00000000 00000000 ................
0006C0 00000000 00000000 00000000 00000000 ................
0006D0 00000000 00000000 00000000 00000000 ................
0006E0 00000000 00000000 00000000 00000000 ................
0006F0 00000000 00000000 00000000 00000000 ................
000700 00000000 00000000 00000000 00000000 ................
000710 00000000 00000000 00000000 00000000 ................
000720 00000000 00000000 00000000 00000000 ................
000730 00000000 00000000 00000000 00000000 ................
000740 00000000 00000000 00000000 00000000 ................
000750 00000000 00000000 00000000 00000000 ................
000760 00000000 00000000 00000000 00000000 ................
000770 00000000 00000000 00000000 00000000 ................
000780 00000000 00000000 00000000 00000000 ................
000790 00000000 00000000 00000000 00000000 ................
0007A0 00000000 00000000 00000000 00000000 ................
0007B0 00000000 00000000 00000000 00000000 ................
0007C0 00000000 00000000 00000000 00000000 ................
0007D0 00000000 00000000 00000000 00000000 ................
0007E0 00000000 00000000 00000000 00000000 ................
0007F0 00000000 00000000 00000000 00000000 ................
C:\vbox\x64test\efi\boot>
C:\vbox\x64test\efi\boot>hexdump bootx64.efi
000000 4D5A9000 03000000 04000000 FFFF0000 MZ..............
000020 00000000 00000000 00000000 00000000 ................
000030 00000000 00000000 00000000 80000000 ................
000040 0E1FBA0E 00B409CD 21B8014C CD215468 ........!..L.!Th
000050 69732070 726F6772 616D2063 616E6E6F is program canno
000060 74206265 2072756E 20696E20 444F5320 t be run in DOS
000070 6D6F6465 2E0D0D0A 24000000 00000000 mode....$.......
000080 50450000 64860200 825F1C64 00000000 PE..d...._.d....
000090 00000000 F0002E02 0B020224 00020000 ...........$....
0000A0 00020000 00000000 00100000 00100000 ................
0000C0 04000000 00000000 05000200 00000000 ................
0000D0 00300000 00040000 E1C20000 0A006001 .0............`.
0000E0 00002000 00000000 00100000 00000000 .. .............
0000F0 00001000 00000000 00100000 00000000 ................
000100 00000000 10000000 00000000 00000000 ................
000110 00200000 14000000 00000000 00000000 . ..............
000120 00000000 00000000 00000000 00000000 ................
000130 00000000 00000000 00000000 00000000 ................
000140 00000000 00000000 00000000 00000000 ................
000150 00000000 00000000 00000000 00000000 ................
000160 00000000 00000000 00000000 00000000 ................
000170 00000000 00000000 00000000 00000000 ................
000180 00000000 00000000 2E746578 74000000 .........text...
000190 60000000 00100000 00020000 00040000 `...............
0001A0 00000000 00000000 00000000 20005060 ............ .P`
0001B0 2E696461 74610000 14000000 00200000 .idata....... ..
0001C0 00020000 00060000 00000000 00000000 ................
0001E0 00000000 00000000 00000000 00000000 ................
0001F0 00000000 00000000 00000000 00000000 ................
000200 00000000 00000000 00000000 00000000 ................
000210 00000000 00000000 00000000 00000000 ................
000220 00000000 00000000 00000000 00000000 ................
000230 00000000 00000000 00000000 00000000 ................
000240 00000000 00000000 00000000 00000000 ................
000250 00000000 00000000 00000000 00000000 ................
000260 00000000 00000000 00000000 00000000 ................
000270 00000000 00000000 00000000 00000000 ................
000280 00000000 00000000 00000000 00000000 ................
000290 00000000 00000000 00000000 00000000 ................
0002A0 00000000 00000000 00000000 00000000 ................
0002B0 00000000 00000000 00000000 00000000 ................
0002C0 00000000 00000000 00000000 00000000 ................
0002D0 00000000 00000000 00000000 00000000 ................
0002E0 00000000 00000000 00000000 00000000 ................
0002F0 00000000 00000000 00000000 00000000 ................
000300 00000000 00000000 00000000 00000000 ................
000310 00000000 00000000 00000000 00000000 ................
000320 00000000 00000000 00000000 00000000 ................
000330 00000000 00000000 00000000 00000000 ................
000340 00000000 00000000 00000000 00000000 ................
000350 00000000 00000000 00000000 00000000 ................
000360 00000000 00000000 00000000 00000000 ................
000370 00000000 00000000 00000000 00000000 ................
000380 00000000 00000000 00000000 00000000 ................
000390 00000000 00000000 00000000 00000000 ................
0003A0 00000000 00000000 00000000 00000000 ................
0003B0 00000000 00000000 00000000 00000000 ................
0003C0 00000000 00000000 00000000 00000000 ................
0003D0 00000000 00000000 00000000 00000000 ................
0003E0 00000000 00000000 00000000 00000000 ................
0003F0 00000000 00000000 00000000 00000000 ................
000400 C3C3C3C3 C3C3C3C3 C3C3C3C3 C3C3C3C3 ................
000410 C3C3C3C3 C3C3C3C3 C3C3C3C3 C3C3C3C3 ................
000420 C3C3C3C3 C3C3C3C3 C3C3C3C3 C3C3C3C3 ................
000430 C3C3C3C3 C3C3C3C3 C3C3C3C3 C3C39090 ................
000440 FFFFFFFF FFFFFFFF 00000000 00000000 ................
000450 FFFFFFFF FFFFFFFF 00000000 00000000 ................
000460 00000000 00000000 00000000 00000000 ................
000470 00000000 00000000 00000000 00000000 ................
000480 00000000 00000000 00000000 00000000 ................
000490 00000000 00000000 00000000 00000000 ................
0004A0 00000000 00000000 00000000 00000000 ................
0004B0 00000000 00000000 00000000 00000000 ................
0004C0 00000000 00000000 00000000 00000000 ................
0004D0 00000000 00000000 00000000 00000000 ................
0004E0 00000000 00000000 00000000 00000000 ................
0004F0 00000000 00000000 00000000 00000000 ................
000500 00000000 00000000 00000000 00000000 ................
000510 00000000 00000000 00000000 00000000 ................
000520 00000000 00000000 00000000 00000000 ................
000530 00000000 00000000 00000000 00000000 ................
000540 00000000 00000000 00000000 00000000 ................
000550 00000000 00000000 00000000 00000000 ................
000560 00000000 00000000 00000000 00000000 ................
000570 00000000 00000000 00000000 00000000 ................
000580 00000000 00000000 00000000 00000000 ................
000590 00000000 00000000 00000000 00000000 ................
0005A0 00000000 00000000 00000000 00000000 ................
0005B0 00000000 00000000 00000000 00000000 ................
0005C0 00000000 00000000 00000000 00000000 ................
0005D0 00000000 00000000 00000000 00000000 ................
0005E0 00000000 00000000 00000000 00000000 ................
0005F0 00000000 00000000 00000000 00000000 ................
000600 00000000 00000000 00000000 00000000 ................
000610 00000000 00000000 00000000 00000000 ................
000620 00000000 00000000 00000000 00000000 ................
000630 00000000 00000000 00000000 00000000 ................
000640 00000000 00000000 00000000 00000000 ................
000650 00000000 00000000 00000000 00000000 ................
000660 00000000 00000000 00000000 00000000 ................
000670 00000000 00000000 00000000 00000000 ................
000680 00000000 00000000 00000000 00000000 ................
000690 00000000 00000000 00000000 00000000 ................
0006A0 00000000 00000000 00000000 00000000 ................
0006B0 00000000 00000000 00000000 00000000 ................
0006C0 00000000 00000000 00000000 00000000 ................
0006D0 00000000 00000000 00000000 00000000 ................
0006E0 00000000 00000000 00000000 00000000 ................
0006F0 00000000 00000000 00000000 00000000 ................
000700 00000000 00000000 00000000 00000000 ................
000710 00000000 00000000 00000000 00000000 ................
000720 00000000 00000000 00000000 00000000 ................
000730 00000000 00000000 00000000 00000000 ................
000740 00000000 00000000 00000000 00000000 ................
000750 00000000 00000000 00000000 00000000 ................
000760 00000000 00000000 00000000 00000000 ................
000770 00000000 00000000 00000000 00000000 ................
000780 00000000 00000000 00000000 00000000 ................
000790 00000000 00000000 00000000 00000000 ................
0007A0 00000000 00000000 00000000 00000000 ................
0007B0 00000000 00000000 00000000 00000000 ................
0007C0 00000000 00000000 00000000 00000000 ................
0007D0 00000000 00000000 00000000 00000000 ................
0007E0 00000000 00000000 00000000 00000000 ................
0007F0 00000000 00000000 00000000 00000000 ................
C:\vbox\x64test\efi\boot>
If you rename loop.efi to bootx64.efi, you will get an
infinite loop as the first instruction.
bootx64.efi as it stands just returns immediately (via
ret) so you will return to your BIOS or a shell may be
executed or something else.
I included a whole lot of "ret" instructions as padding
so that you can zap your own machine code in to do
further testing.
I didn't use "nop" because the assembler pads with nop.
If you zap your own machine code, the required checksum
of the PE executable will change - I don't know if all/any
UEFI actually check that. The location of the checksum
C:\vbox\x64test\efi\boot>fc /b loop.efi bootx64.efi
Comparing files loop.efi and BOOTX64.EFI
00000088: F2 82
00000089: 5E 5F
000000D8: 79 E1
000000D9: FD C2
00000400: EB C3
00000401: FE C3
C:\devel\pdos\pdpclib\xxx3>fc /b \vbox\x64test\efi\boot\bootx64.efi bootx64.efi
Comparing files \VBOX\X64TEST\EFI\BOOT\bootx64.efi and BOOTX64.EFI
00000088: 82 49
00000089: 5F 64
000000D8: E1 A8
000000D9: C2 C7
Maybe the timestamp is included in the checksum.
So take a guess.
You'll presumably need to disable secure boot in the BIOS too.
1. I'm not sure you will be in ring 0. Maybe try writing to the serial
port - I don't think you can write to b8000 because I think you
will be in graphics mode.
2. The loop will terminate after a while (minutes?) because of a
watchdog timer. There is a UEFI call to disable the watchdog.
I'm not sure you will need to do that if your bootloader does
something substantial.
3. As far as I know, most people do a UEFI call to "exit boot
services" before they start doing something substantial.
I don't know if you can bypass that.
So you may need to do 0, 1 or 2 UEFI calls.
C:\devel\pdos\pdpclib>grep -i exit efi.h
efi.h: void *Exit;
efi.h: void *ExitBootServices;
C:\devel\pdos\pdpclib>grep -i watchdog efi.h
efi.h: typedef EFI_STATUS (EFIAPI *EFI_SET_WATCHDOG_TIMER) (IN UINTN Timeout, IN UINT64 WatchdogCode, IN UINTN DataSize, IN CHAR16 *WatchdogData);
efi.h: EFI_SET_WATCHDOG_TIMER SetWatchdogTimer;
C:\devel\pdos\pdpclib>
But I've only tested the watchdog.
BFN. Paul.

Rick C. Hodgin

2023-03-23 18:17:14 UTC

Post by wolfgang kern
with a new assembled PC I get a main-board with a CD that contains only
a few drivers for windoze loonix and Mac, nothing boot able on it.
the HDs are all empty, not formatted in any usable mode.

I have this thought, and then I'll bow out:

For a custom OS, it's enough to support the 1990s-based 80386 and/or 80486 systems and later up through probably the AMD-based Pentium equivalents, and even those systems with just a few MB of RAM. We forget what powerful software we had back in those days with limited resources. They just weren't GUI, or didn't use complex classes and objects, but more programming in C or assembly.

Those custom OSes will need only basic drivers for basic motherboard devices, floppy, hard disk, keyboard, mouse, video, and most importantly: networking. USB would be nice, as would things like SD cards and more modern storage as they're easy to transport. But I think it would be fine to stick with even older technologies as those are fairly easy to cobble together and get up and running with limited resources / labs / equipment.

IMO, for the world that's coming, having the ability to do any type of basic computation, and to be able to communicate with others using even very basic software, will be sufficient in that environment. Using DOS 6.x and QEMM would probably be best, along with old Novell networking, because many drivers for many compatible hardware devices existed into the late 90s.

A new era is almost upon us. It's time to prepare for it. Stock up on Ethernet cables, old compatible network cards, old PCs, etc.

And most importantly, I advise making preparations at the foot of the cross before doing any of this, and then concerning yourselves with all these other places after that.

If any of you wish to contact me, feel free to do so.

--
Rick C. Hodgin

T. Ment

2023-03-23 22:32:41 UTC

Post by Rick C. Hodgin
Using DOS 6.x and QEMM would probably be best,

I use QEMM but it's unstable. MS himem.sys is better. Maybe that's why
QDK went out of business and MS carried on.

Post by Rick C. Hodgin
along with old Novell networking,

They went out of business too.

DOS Client32 with TCPIP.NLM is stable for me, provided I keep the config
simple. Trying to add extra protcols crashes it.

You need their developer TCP libs to write any TCP apps. Try and find
them now. Good luck.

muta...@gmail.com

2023-03-23 22:51:18 UTC

Post by Rick C. Hodgin
Using DOS 6.x and QEMM would probably be best,

I use QEMM but it's unstable. MS himem.sys is better. Maybe that's why
QDK went out of business and MS carried on.

Post by Rick C. Hodgin
along with old Novell networking,

They went out of business too.
DOS Client32 with TCPIP.NLM is stable for me, provided I keep the config
simple. Trying to add extra protcols crashes it.
You need their developer TCP libs to write any TCP apps. Try and find
them now. Good luck.

Good to see you back too.

Does Freedos not have everything you need?

What's the advantage of Microsoft and scrounging for
old software?

BFN. Paul.

T. Ment

2023-03-23 23:38:55 UTC

Post by ***@gmail.com
Does Freedos not have everything you need?

It's more than I need.

Post by ***@gmail.com
What's the advantage of Microsoft and scrounging for
old software?

Installing MSDOS 6.22 and Windows 3.1 is quick and easy. I don't have
any agenda about licensing. It's just a hobby for me.

Grant Taylor

2023-03-24 00:29:05 UTC

Post by T. Ment
Installing MSDOS 6.22 and Windows 3.1 is quick and easy. I don't have
any agenda about licensing. It's just a hobby for me.

You might not have an agenda about the licensing. But that doesn't
change the fact that both MS-DOS and Microsoft Windows 3.1 were ~> still
are licensed products.

/Technically/ you should have a license to use them.

--
Grant. . . .
unix || die

T. Ment

2023-03-24 00:38:21 UTC

Post by Grant Taylor
You might not have an agenda about the licensing. But that doesn't
change the fact that both MS-DOS and Microsoft Windows 3.1 were ~> still
are licensed products.
/Technically/ you should have a license to use them.

Who said I don't?

muta...@gmail.com

2023-03-24 00:53:14 UTC

You might not have an agenda about the licensing. But that doesn't
change the fact that both MS-DOS and Microsoft Windows 3.1 were ~> still
are licensed products.
/Technically/ you should have a license to use them.

Who said I don't?

It's not a good solution if e.g. your house burns down so you
lose any evidence you had a license and need to buy a new
PC.

Or if you want to help a friend set up a PC.

Or start a business.

If Freedos is more than you need, why not create a trimmed-down
installation of it?

Incidentally, I have such a trimmed down installation myself.

Along with what is apparently the only surviving copy of
Open Watcom 1.6 on the net.

BFN. Paul.

Grant Taylor

2023-03-24 16:06:41 UTC

Post by T. Ment
Who said I don't?

Nobody.

--
Grant. . . .
unix || die

muta...@gmail.com

2023-03-23 22:48:35 UTC

Post by Rick C. Hodgin
FWIW, I miss this group.

No-one asked you to leave - people just asked you to
stop posting religious stuff.
Alternatively, you can answer questions people ask,
and refrain from posting religious stuff that no-one
at all asked for. (with the possible exception of
this post).

Post by Rick C. Hodgin
For a custom OS, it's enough to support the 1990s-based
80386 and/or 80486 systems and later up through
probably the AMD-based Pentium equivalents, and
even those systems with just a few MB of RAM.

What do you mean "enough"? You can't easily buy
such computers.

Post by Rick C. Hodgin
We forget what powerful software we had back in
those days with limited resources. They just weren't
GUI, or didn't use complex classes and objects, but
more programming in C or assembly.

Absolutely. And commercial enterprises ran on
mainframes with limited memory too. They weren't
mere toys. It was an entire industry.

Planes were developed with just a few K of memory
before that too. Not so much "before that" as "in the
beginning".

Post by Rick C. Hodgin
Those custom OSes will need only basic drivers for basic
motherboard devices, floppy, hard disk, keyboard, mouse,
video, and most importantly: networking. USB would be
nice, as would things like SD cards and more modern
storage as they're easy to transport. But I think it would
be fine to stick with even older technologies as those are
fairly easy to cobble together and get up and running with
limited resources / labs / equipment.

Older technologies like floppy disks? Is the media still sold?

Post by Rick C. Hodgin
IMO, for the world that's coming, having the ability to do any
type of basic computation, and to be able to communicate
with others using even very basic software, will be sufficient
in that environment. Using DOS 6.x and QEMM would probably
be best, along with old Novell networking, because many drivers
for many compatible hardware devices existed into the late 90s.

Ok, in the past I have postulated nuclear war and time travel
and some asshole locking you up in the basement as potential
reasons you might be forced to work with limited memory.

And the guy who wrote "Collapse OS" thinks that is a realistic
situation too.

Can you explain the situation you apparently believe in, that
attracts you to DOS 6? You already mentioned 80386. Why
not a 32-bit OS (early Linux or anything else)?

Please limit any mention of God or Jesus to less than 100
words. But I'm interested in what technology remains after
God and Jesus have combined forces to unleash their
wrath.

Post by Rick C. Hodgin
A new era is almost upon us. It's time to prepare for it. Stock
up on Ethernet cables, old compatible network cards, old PCs, etc.

It's fantastic to see a real live computer "prepper".

Actually maybe I'm one of those too. I have bought a
lot of computers, but that's for testing, not really
for prepping hardware. I've been prepping software.

Post by Rick C. Hodgin
And most importantly, I advise making preparations at the
foot of the cross before doing any of this,

That I'm not interested in doing.

I'm happy to go to Hell. But before I do, I'm interested
in helping those who go to wherever you're going have
some suitable software for their journey. Which is what
PDOS is.

Post by Rick C. Hodgin
and then concerning yourselves with all these other places after that.

What other places?

Post by Rick C. Hodgin
If any of you wish to contact me, feel free to do so.

Consider yourself contacted and on-topic.

BFN. Paul.

muta...@gmail.com

2023-03-23 22:27:18 UTC

Post by wolfgang kern
So what is the UEFI equivalent to INT13 and how can I make it boot my
old stuff from either CD or USB ?

As mentioned, if all you need is a bit of code loaded,
you don't need to anything at all except plonk a
bootx64.efi into \efi\boot of a FAT-formatted disk.

with a new assembled PC I get a main-board with a CD that contains only
a few drivers for windoze loonix and Mac, nothing boot able on it.
the HDs are all empty, not formatted in any usable mode.

You have an existing PC, right?

So ignore the CD that came with the new PC.

On your old PC create a FAT-formatted USB stick,
or Joliet CD.

With the bootx64.ef on it.

And then boot from CD or USB stick (you may need to go
into the BIOS settings to choose to allow booting from
those devices).

Post by wolfgang kern
So it would need me to buy a win11-DVD which I wont ever do.
any suggestion ? I'd need a boot able formatting tool first.

What do you currently use?

If you want to use someone else's OS that is capable of
formatting a disk and also capable of booting on UEFI,
and it can't be Windows, and it can't be Linux, I am not
familiar with that market. FreeBSD? PDOS/386 only
works with a BIOS system currently. PDOS-generic will
do UEFI but it is still proof-of-concept. It will basically
reuse the existing code though, so it shouldn't be a
big deal. The big deal is me trying to get it to do
unusual things like running 32-bit code in 64-bit mode
before taking it out of POC.

Post by wolfgang kern
Next demand would be a tool which can create folders.

That tool is normally called an OS, but there are some
tools called "mtools", and a fairly recent public domain
version too, that do that to a disk image.

On PDOS/386 I can make that disk image an actual
raw disk, which is what I use for formatting.

The tools are included on the PDOS/386 image.

They're written in C90 so you can run them on any other
system you can find (after recompilation).

Post by wolfgang kern
And then such a tool must be able to copy my OS-boot-image
into a bootx64.efi perhaps at a certain offset dunno yet how
.efi files were organized (I may find some info on the net)
but the hex-dumps below show that you created M$ PE-files.

Yes, there is no choice. You need to create a PE executable.

It's just a normal PE executable, with the sole exception that
the subsystem number is set to 10. So that's one byte that
needs to change (plus the checksum, as that byte is included).

The linker will do this for you.

It's unclear to me what restrictions you are introducing,
or why, but I know of two linkers that produce 64-bit
PE files, both running under Windows, and neither
supplied with any version of PDOS. But if you just
want a single binary with 100k of "ret" or "nop" or x'00',
I can provide that for you.

Post by wolfgang kern
one more problem: my OS-images incl. boot-part are just a bunch of
consecutive sectors (can be seen and stored as .bin files)
no header, no redirection, very own checksum algo.

That can be embedded in the PE file. It's just code.

But you would need to take care of relocation yourself
if you are just zapping bytes instead of using a linker.

Post by ***@gmail.com
# So for testing, comment out the jmp and add 2 rets

dup 64 ret ;wouldn't one be enough?

I didn't know the syntax.

Post by wolfgang kern
;where would this return go to ?

The BIOS.

That above code is entered (from UEFI) via a call.

There will be 2 parameters (in registers, not on the
stack - that's the Microsoft 64-bit calling convention).

You would normally never return.

I returned just for illustration purposes.

And looped for illustration purposes too.

Post by wolfgang kern
so I assume the first 64 bytes are special purpose ?

No. If it was 64 bytes, that was just an accident. I just
typed in ret and then used micro-emacs and did ctrl-y
a large number of times so that you had some space
to zap.

Post by wolfgang kern
[...]there is no assembler :) and rare ever will be.

You want to enter machine code, right?

So all I did was make space for testing purposes.
I used serial port as an example. I am guessing you
can send a single byte to the serial port in that
amount of space.

BFN. Paul.

wolfgang kern

2023-03-24 02:48:51 UTC

Post by ***@gmail.com
You have an existing PC, right?

yes I post from a windoze_10 here.

Post by ***@gmail.com
So ignore the CD that came with the new PC.
On your old PC create a FAT-formatted USB stick,
or Joliet CD.

all my older PCs died already, format options on my current one don't
imply old style FAT32, only NTFS and EXFAT.

Post by ***@gmail.com
With the bootx64.ef on it.

I see that it would need to have all my disk-functions already in this
boot.efi. My installer have to create partitions and format these before
the OS can be copied to it.
So I'd had to enter my whole start image (512MB) into this PE.

Post by ***@gmail.com
And then boot from CD or USB stick (you may need to go
into the BIOS settings to choose to allow booting from
those devices).

Post by wolfgang kern
So it would need me to buy a win11-DVD which I wont ever do.
any suggestion ? I'd need a boot able formatting tool first.

What do you currently use?

win 10 on MSI Ryzen5. cannot boot from USB.

Post by ***@gmail.com
If you want to use someone else's OS that is capable of
formatting a disk and also capable of booting on UEFI,
and it can't be Windows, and it can't be Linux, I am not
familiar with that market. FreeBSD? PDOS/386 only
works with a BIOS system currently. PDOS-generic will
do UEFI but it is still proof-of-concept. It will basically
reuse the existing code though, so it shouldn't be a
big deal. The big deal is me trying to get it to do
unusual things like running 32-bit code in 64-bit mode
before taking it out of POC.

Post by wolfgang kern
Next demand would be a tool which can create folders.

That tool is normally called an OS, but there are some
tools called "mtools", and a fairly recent public domain
version too, that do that to a disk image.
On PDOS/386 I can make that disk image an actual
raw disk, which is what I use for formatting.
The tools are included on the PDOS/386 image.
They're written in C90 so you can run them on any other
system you can find (after recompilation).

I can't re/compile any C-source. there is no C on any of my machines.

there is no linker on my PC.
I have several options to create almost empty PE with
RosASM, FASM, NASM, but they aren't 64-bit ?

Post by ***@gmail.com
It's unclear to me what restrictions you are introducing,
or why, but I know of two linkers that produce 64-bit
PE files, both running under Windows, and neither
supplied with any version of PDOS. But if you just
want a single binary with 100k of "ret" or "nop" or x'00',
I can provide that for you.

Thanks, once I figure the difference I might be able to modify.

That can be embedded in the PE file. It's just code.
But you would need to take care of relocation yourself
if you are just zapping bytes instead of using a linker.

There are no relocation needs in my code.
...

Post by ***@gmail.com
That above code is entered (from UEFI) via a call.
There will be 2 parameters (in registers, not on the
stack - that's the Microsoft 64-bit calling convention).

OK fine, tell me more :)
where can I find a list of available UEFI functions ?
what I found on the net was just small talk w/o sense.

Post by wolfgang kern
[...]there is no assembler :) and rare ever will be.

You want to enter machine code, right?

yes, I stored my OS image (512MB) on USB and on CD.

Post by ***@gmail.com
So all I did was make space for testing purposes.

OK.

Post by ***@gmail.com
I used serial port as an example. I am guessing you
can send a single byte to the serial port in that
amount of space.

there might not be any serial port an new machines.
__
wolfgang

muta...@gmail.com

2023-03-24 03:51:55 UTC

Post by ***@gmail.com
You have an existing PC, right?

yes I post from a windoze_10 here.

Post by ***@gmail.com
So ignore the CD that came with the new PC.
On your old PC create a FAT-formatted USB stick,
or Joliet CD.

all my older PCs died already, format options on my current one don't
imply old style FAT32, only NTFS and EXFAT.

Windows 10 supports formatting as FAT32 and in
fact, FAT12/16. And MBR too.

Right-click on the start menu and choose "disk
management". Although normally I have created
virtual hard disks. And when I have a virtual hard
disk I like I will then burn that to a USB stick using
Win32 disk imager.

Post by ***@gmail.com
With the bootx64.ef on it.

I see that it would need to have all my disk-functions already in this
boot.efi. My installer have to create partitions and format these before
the OS can be copied to it.

You're talking about after you have a computer with just
your OS on it, right?

You can presumably run applications under your OS, so
your installer/disk formatter would be one such
application obeying your OS's rules.

Post by wolfgang kern
So I'd had to enter my whole start image (512MB) into this PE.

No reason it needs to be designed that way.

My bootx64.efi is 48k in size. It then launches my real
OS (PDOS-generic) which is 73k and will start growing
as I add more functionality to it. It won't grow very
much though, as I don't have much functionality I
want to add.

Post by ***@gmail.com
And then boot from CD or USB stick (you may need to go
into the BIOS settings to choose to allow booting from
those devices).

Post by wolfgang kern
So it would need me to buy a win11-DVD which I wont ever do.
any suggestion ? I'd need a boot able formatting tool first.

What do you currently use?

win 10 on MSI Ryzen5. cannot boot from USB.

I find that hard to believe.

You probably just haven't put a bootable (\EFI\BOOT\BOOTX64.EFI)
USB stick into the machine so that it is recognized and presented
to you as a boot option.

Are you able to provide a photo of your BIOS screens?

Post by ***@gmail.com
On PDOS/386 I can make that disk image an actual
raw disk, which is what I use for formatting.
The tools are included on the PDOS/386 image.
They're written in C90 so you can run them on any other
system you can find (after recompilation).

I can't re/compile any C-source. there is no C on any of my machines.

I can provide generated 80386 assembler for a
C compiler.

The assembler format is AT&T though.

I may be able to switch to intel format if necessary.

But the compiler itself will generate AT&T syntax
so you will need an AT&T assembler.

I can potentially provide intel format for that too.

there is no linker on my PC.

Ok, now we're talking about 64-bit tools. I don't have the
infrastructure for that, so I can't currently provide assembler
code for a 64-bit linker.

Other people probably can though.

Post by wolfgang kern
I have several options to create almost empty PE with
RosASM, FASM, NASM, but they aren't 64-bit ?

64-bit is new to me, but we're now back at that situation
we were discussing about running 32-bit code in x64
mode.

You can presumably get RosASM or whatever to generate
machine code, and if you select the instructions properly
so that they are 64-bit clean, you can copy the machine
code onto that PE executable I sent you.

(or I can provide a bigger executable, zipped).

Post by ***@gmail.com
That above code is entered (from UEFI) via a call.
There will be 2 parameters (in registers, not on the
stack - that's the Microsoft 64-bit calling convention).

OK fine, tell me more :)

The calling convention?

Post by wolfgang kern
where can I find a list of available UEFI functions ?
what I found on the net was just small talk w/o sense.

Ok, bear in mind that I started UEFI development without
documentation. What I instead did was ask someone for
offsets. And I had previously looked at a "hello world"
on github.

There was quite a saga to get it working, but 32-bit
didn't work.

64-bit I couldn't get to work, and I gave up.

Some months later someone else came along with an
interest in UEFI and fleshed out the 32-bit some more,
plus told me that to get the 64-bit working I just needed
to drop the packed_struct option. As far as I can tell
they just figured that out by experimentation.

So, I can't vouch for any documentation.

However, after that I had an argument with someone
in the PDOS discord server about raw disk access
(LBA) and as part of that discussion, they gave me
this link:

https://discordapp.com/channels/850670546903498752/850670546903498756/1081240235666575440

https://uefi.org/specs/UEFI/2.10/13_Protocols_Media_Access.html#block-i-o-protocol

I did click on the link at the time to see what they were
talking about, but I haven't clicked on it since (or now).

Post by wolfgang kern
[...]there is no assembler :) and rare ever will be.

You want to enter machine code, right?

yes, I stored my OS image (512MB) on USB and on CD.

That size means that it is a disk image, not just your OS.

Post by ***@gmail.com
I used serial port as an example. I am guessing you
can send a single byte to the serial port in that
amount of space.

there might not be any serial port an new machines.

Sure. But you can buy a new PC with a serial port, right?

If neither the display nor a serial port is available to
test some machine code, I don't know what other
option you might have to test sign of life.

This is a problem in my mind - they are making it
difficult to write and understand code.

I don't suppose you can switch the num lock key
on or something via direct hardware manipulation?

BTW, you didn't explain what the restriction is.

What software are you willing to download?

Nothing?

Nothing except assembler source?

BFN. Paul.

wolfgang kern

2023-03-24 08:21:53 UTC

Post by wolfgang kern
all my older PCs died already, format options on my current one don't
imply old style FAT32, only NTFS and EXFAT.

Windows 10 supports formatting as FAT32 and in
fact, FAT12/16. And MBR too.
Right-click on the start menu and choose "disk
management". Although normally I have created
virtual hard disks. And when I have a virtual hard
disk I like I will then burn that to a USB stick using
Win32 disk imager.

doesn't show FAT32 for CD nor USB.
I might need to buy me an older second hand PC for this,
but what I found so far is either damaged or not suitable.

Post by ***@gmail.com
With the bootx64.ef on it.

I see that it would need to have all my disk-functions already in this
boot.efi. My installer have to create partitions and format these before
the OS can be copied to it.

You're talking about after you have a computer with just
your OS on it, right?

no, there could be also other boot partitions.

Post by ***@gmail.com
You can presumably run applications under your OS,

no, for safety reason never ever run foreign code,
only scripts (written by me or members) were "executed".

Post by ***@gmail.com
so your installer/disk formatter would be one such
application obeying your OS's rules.

even the format tool is needed before the OS can be loaded, it's part of
the OS anyway.

Post by wolfgang kern
So I'd had to enter my whole start image (512MB) into this PE.

No reason it needs to be designed that way.

My OS is one consecutive block, I wont tear it apart. while the first
few sectors were just there to check "where I am" and to load the rest
with INT13, these functions are reusable and therefor remain in RAM.

Post by ***@gmail.com
And then boot from CD or USB stick (you may need to go
into the BIOS settings to choose to allow booting from
those devices).

You probably just haven't put a bootable (\EFI\BOOT\BOOTX64.EFI)
USB stick into the machine so that it is recognized and presented
to you as a boot option.

the problem still is to create a UEFI64 USB

Post by ***@gmail.com
Are you able to provide a photo of your BIOS screens?

not yet.
...

Post by wolfgang kern
I can't re/compile any C-source. there is no C on any of my machines.

I can provide generated 80386 assembler for a
C compiler.

thank you, but NO thank you! C wont ever enter my world.

Post by ***@gmail.com
Yes, there is no choice. You need to create a PE executable.

...

Post by wolfgang kern
I have several options to create almost empty PE with
RosASM, FASM, NASM, but they aren't 64-bit ?

there is much more than instructions to consider, the whole environment
changes a lot between segmented PM32 and LM64.

Post by ***@gmail.com
you can copy the machine
code onto that PE executable I sent you.

I kept your hex dump to see what it does.

Post by ***@gmail.com
There will be 2 parameters (in registers, not on the
stack - that's the Microsoft 64-bit calling convention).

OK fine, tell me more :)

The calling convention?

already know that, stored the info in my library (text).

Post by wolfgang kern
where can I find a list of available UEFI functions ?
what I found on the net was just small talk w/o sense.

...

Post by ***@gmail.com
So, I can't vouch for any documentation.

...

Post by ***@gmail.com
https://uefi.org/specs/UEFI/2.10/13_Protocols_Media_Access.html#block-i-o-protocol

THX, this one was what I searched for, no wonder that I didn't find it.

Post by wolfgang kern
[...]there is no assembler :) and rare ever will be.

You want to enter machine code, right?

yes, I stored my OS image (512MB) on USB and on CD.

That size means that it is a disk image, not just your OS.

No, I decided for this limit for boot speed reason.
My KESYS was ready after 2 seconds (after BIOS finished)

Post by ***@gmail.com
I used serial port as an example. I am guessing you
can send a single byte to the serial port in that
amount of space.

there might not be any serial port an new machines.

Sure. But you can buy a new PC with a serial port, right?

probable not. New boards don't have such anymore.

Post by ***@gmail.com
If neither the display nor a serial port is available to
test some machine code, I don't know what other
option you might have to test sign of life.

screen will work after figure where the graphic RAM reside.

Post by ***@gmail.com
This is a problem in my mind - they are making it
difficult to write and understand code.
I don't suppose you can switch the num lock key
on or something via direct hardware manipulation?

yes of course I can. My OS is as hardware near as the CPU is :)

Post by ***@gmail.com
BTW, you didn't explain what the restriction is.

in which context ?

Post by ***@gmail.com
What software are you willing to download?

that depends on just one thing: usability.
__
wolfgang

muta...@gmail.com

2023-03-24 10:25:05 UTC

Post by ***@gmail.com
Right-click on the start menu and choose "disk
management". Although normally I have created
virtual hard disks. And when I have a virtual hard
disk I like I will then burn that to a USB stick using
Win32 disk imager.

doesn't show FAT32 for CD nor USB.

What options do you see for a virtual hard disk?

I just went to create a virtual hard disk as above.

I made it fixed size (500 MB).

I initialized it as MBR.

I created a new simple volume and the formatting
options included FAT (which means 12/16), FAT32
and NTFS.

I haven't retested a direct to USB stick/CD.

But that image can definitely be burned to USB stick.

Post by wolfgang kern
I might need to buy me an older second hand PC for this,
but what I found so far is either damaged or not suitable.

I am using a modern machine and Windows 10. It works fine.

Post by ***@gmail.com
You probably just haven't put a bootable (\EFI\BOOT\BOOTX64.EFI)
USB stick into the machine so that it is recognized and presented
to you as a boot option.

the problem still is to create a UEFI64 USB

Unless there is a glitch in the Matrix that makes Windows 10
work for me (in the Philippines), but not you (in Germany?),
it should be simple.

Post by ***@gmail.com
Sure. But you can buy a new PC with a serial port, right?

probable not. New boards don't have such anymore.

Really?

I thought the thing was there, it just didn't have anything
attached.

Post by ***@gmail.com
If neither the display nor a serial port is available to
test some machine code, I don't know what other
option you might have to test sign of life.

screen will work after figure where the graphic RAM reside.

Ok.

Post by ***@gmail.com
I don't suppose you can switch the num lock key
on or something via direct hardware manipulation?

yes of course I can. My OS is as hardware near as the CPU is :)

Ok, so maybe you can do it within the space available.

Post by ***@gmail.com
BTW, you didn't explain what the restriction is.

in which context ?

What you've started answering below.

Post by ***@gmail.com
What software are you willing to download?

that depends on just one thing: usability.

Ok, so create a VHD (MBR/FAT-anything) and then burn to
physical USB stick with this:

https://sourceforge.net/projects/win32diskimager/

Then I want to see evidence of your BIOS not accepting
booting from it.

Do you have the name of your BIOS or anything so that
I can see documentation? I already searched for your
PC type but didn't see a manual.

BFN. Paul.

wolfgang kern

2023-03-24 17:19:34 UTC

Post by ***@gmail.com
What options do you see for a virtual hard disk?

Never tried because there are about 20 new HDs on my shelf.

Post by ***@gmail.com
I just went to create a virtual hard disk as above.
I made it fixed size (500 MB).
I initialized it as MBR.
I created a new simple volume and the formatting
options included FAT (which means 12/16), FAT32
and NTFS.

I wont fool myself with fake virtualization.

Post by ***@gmail.com
I haven't retested a direct to USB stick/CD.
But that image can definitely be burned to USB stick.

Post by wolfgang kern
I might need to buy me an older second hand PC for this,
but what I found so far is either damaged or not suitable.

I am using a modern machine and Windows 10. It works fine.

Post by ***@gmail.com
You probably just haven't put a bootable (\EFI\BOOT\BOOTX64.EFI)
USB stick into the machine so that it is recognized and presented
to you as a boot option.

the problem still is to create a UEFI64 USB

Unless there is a glitch in the Matrix that makes Windows 10
work for me (in the Philippines), but not you (in Germany?),
it should be simple.

my win10 is restricted because of several hardware changes.
".at" isn't Germany.

why test a non present Serial port, even if one exists there is no
indication if your code work or not.

Post by wolfgang kern
screen will work after figure where the graphic RAM reside.

Ok.

Post by ***@gmail.com
I don't suppose you can switch the num lock key
on or something via direct hardware manipulation?

yes of course I can. My OS is as hardware near as the CPU is :)

Ok, so maybe you can do it within the space available.

you got something wrong here :)
I'm still in the phase of collecting information rather than gadget
around in the wild yet.

Post by ***@gmail.com
What software are you willing to download?

I may download the latest BKDG and a new South bridge manual from AMD.
then I will convert all the info found in UEFI.org into a non-C format.
__
wolfgang

Scott Lurndal

2023-03-24 18:40:55 UTC

Post by wolfgang kern
my win10 is restricted because of several hardware changes.
".at" isn't Germany.

I always wondered why they chose .at rather than .Ös :-)

wolfgang kern

2023-03-24 21:03:24 UTC

Post by wolfgang kern
my win10 is restricted because of several hardware changes.
".at" isn't Germany.

I always wondered why they chose .at rather than .Ös :-)

there weren't Ö available so first idea was the french .aut
but decided for .at to have more difference to .aus
local cars are marked with an "A".
__
wolfgang

muta...@gmail.com

2023-03-24 22:53:08 UTC

Post by ***@gmail.com
What options do you see for a virtual hard disk?

Never tried because there are about 20 new HDs on my shelf.

I wont fool myself with fake virtualization.

It's not fake. After you burn the VHD to a real USB
stick, you have a non-fake OS on the hardware.

Post by ***@gmail.com
Unless there is a glitch in the Matrix that makes Windows 10
work for me (in the Philippines), but not you (in Germany?),
it should be simple.

my win10 is restricted because of several hardware changes.

Ok, that's an interesting restriction. But you still haven't
proven that it applies here.

And even if it does - there is no way to create a FAT(12/16)
or FAT32 VHD or USB stick, there is still another option.

There are a number of VHDs available on my website
(pdos.org).

If you are able to run Win32 disk imager, then burning
any of them to USB stick will give you a FAT disk.

You will then be able to use Windows to create the
EFI\BOOT and plonk in the bootx64.efi

Alternatively you own OS may be capable of doing the
same thing - reading a VHD on some sort of hard disk
that you are able to get to work, and writing sectors to
a USB stick.

Do you own a smartphone? They can format a USB
disk as FAT too. I bought a tablet for 999 pesos that
can do that too. New. An old model of course. That's
under US$20.

Post by wolfgang kern
".at" isn't Germany.

Your email address isn't visible to me on google groups,
just your name.

Post by wolfgang kern
why test a non present Serial port, even if one exists there is no
indication if your code work or not.

No, it will be an indication that your replacement code is
running in ring 0 or whatever such that it can write to an
I/O port.

Post by ***@gmail.com
I don't suppose you can switch the num lock key
on or something via direct hardware manipulation?

yes of course I can. My OS is as hardware near as the CPU is :)

Ok, so maybe you can do it within the space available.

you got something wrong here :)
I'm still in the phase of collecting information rather than gadget
around in the wild yet.

This is information. How much assembler code is
required to switch on num lock? Does switching on
num lock require ring 0? Are you thus in ring 0?

BFN. Paul.

wolfgang kern

2023-03-25 05:26:29 UTC

On 24/03/2023 23:53, ***@gmail.com wrote:
...

Post by wolfgang kern
you got something wrong here :)
I'm still in the phase of collecting information rather than gadget
around in the wild yet.

This is information. How much assembler code is
required to switch on num lock?

four byte plus a 5mS delay.

Post by ***@gmail.com
Does switching on num lock require ring 0?

No, it depends on IOPL.

Post by ***@gmail.com
Are you thus in ring 0?

success on I/O access doesn't tell you are in PL0,
you may run PL2 with IOPL3

to check if I/O access doesn't crash try:

E4 20 IN AL,[0x20]

use any privileged instruction for a PL0 test, there are many,
look for ones that wont cause any trouble if succeed.
__
wolfgang

muta...@gmail.com

2023-03-25 07:10:42 UTC

Post by wolfgang kern
use any privileged instruction for a PL0 test, there are many,
look for ones that wont cause any trouble if succeed.

Ok, that's a good idea.

So using the 2 executables I gave you, you can do:

1. privileged instruction followed by loop
2. privileged instruction followed by ret

and if they behave differently - in fact, identically to
what I sent - then you will have the ring 0 answer
for at least one PC.

Otherwise you will need to find the appropriate UEFI
call. You seem to be doing that already (ie reading
the (presumably copyrighted) documentation), which
is a more robust way of doing things. That's where the
contract exists between you and the computer.

I will refer to such documentation at the point where
someone claims my existing code is non-compliant
and points me to chapter and verse.

BFN. Paul.

wolfgang kern

2023-03-25 08:34:36 UTC

Post by wolfgang kern
use any privileged instruction for a PL0 test, there are many,
look for ones that wont cause any trouble if succeed.

Ok, that's a good idea.
1. privileged instruction followed by loop
2. privileged instruction followed by ret

Not me. I already said that I still collect info.
until I get a pristine clear picture of the whole story.

why you don't get it: _I can't/wont use your files_

I have only one working and eleven dead PCs here,
and I need this one for posting and currently for games.

So you can do this tests and report what you found.

Post by ***@gmail.com
and if they behave differently - in fact, identically to
what I sent - then you will have the ring 0 answer
for at least one PC.
Otherwise you will need to find the appropriate UEFI
call. You seem to be doing that already (ie reading
the (presumably copyrighted) documentation), which
is a more robust way of doing things. That's where the
contract exists between you and the computer.
I will refer to such documentation at the point where
someone claims my existing code is non-compliant
and points me to chapter and verse.

Yeah, this UEFI.org docs are as weird or even more than C is.
may take me a while to navigate through that and convert all
into human readable patterns (make it a sandpile.org format)
__
wolfgang

muta...@gmail.com

2023-03-25 09:10:09 UTC

Post by ***@gmail.com
1. privileged instruction followed by loop
2. privileged instruction followed by ret

Not me. I already said that I still collect info.
until I get a pristine clear picture of the whole story.
why you don't get it: _I can't/wont use your files_

That's the first I've heard that. I thought the barrier
was creating a FAT USB stick and I thought we
were trying to solve that problem.

Post by wolfgang kern
I have only one working and eleven dead PCs here,
and I need this one for posting and currently for games.

You can't afford a new computer so that you can do
experiments?

Again - around US$200 new here. I paid 9200 pesos
for the computer I'm using right now. 64 GB hard
disk (SSD/EMC?) of which about half is used for
Windows. Maybe a bit more. So space does get a
bit tight, but I've managed to do my development
just fine regardless. 4 GB RAM.

Post by wolfgang kern
So you can do this tests and report what you found.

If you give me a specific instruction to test (ie something
that will assemble with the mingw assembler), I can do
so. But if you're reading the docs anyway, you'll
probably get the answer there.

BFN. Paul.

wolfgang kern

2023-03-25 11:21:47 UTC

Post by wolfgang kern
why you don't get it: _I can't/wont use your files_

That's the first I've heard that. I thought the barrier
was creating a FAT USB stick and I thought we
were trying to solve that problem.

Post by wolfgang kern
I have only one working and eleven dead PCs here,
and I need this one for posting and currently for games.

You can't afford a new computer so that you can do
experiments?

I don't buy M$ prepared PCs, I bought one 1980, and I always assembled
machines individual since then usually just to fit client's demand.

[I don't have any problems with money :)]

Post by wolfgang kern
So you can do this tests and report what you found.

If you give me a specific instruction to test (ie something
that will assemble with the mingw assembler), I can do so.

dunno mingw, but it may be able to use "db HEX" like other ASM do.

33 C9 XOR ecx,ecx
B1 1B MOV CL.0x1B ;make RCXX==0x1B (returns APIC_BASE on AMDs)
0F 32 RDMSR ;may crash if this MSR doesn't exist or CPL>0

or an easier one:

0F 09 WBINV ;alters no registers crash if CPL>0.

Post by ***@gmail.com
But if you're reading the docs anyway, you'll
probably get the answer there.

haven't found anything mentioned about start condition in these docs so
far.
__
wolfgang

muta...@gmail.com

2023-03-25 12:32:45 UTC

Post by wolfgang kern
why you don't get it: _I can't/wont use your files_

That's the first I've heard that. I thought the barrier
was creating a FAT USB stick and I thought we
were trying to solve that problem.

Post by wolfgang kern
I have only one working and eleven dead PCs here,
and I need this one for posting and currently for games.

You can't afford a new computer so that you can do
experiments?

I don't buy M$ prepared PCs, I bought one 1980, and I always assembled
machines individual since then usually just to fit client's demand.
[I don't have any problems with money :)]

Ok, it's still unclear to me what (artificial) constraints
you are placing, and why, but they are clearly genuine
constraints, and that is an interesting use case scenario.

Post by wolfgang kern
So you can do this tests and report what you found.

If you give me a specific instruction to test (ie something
that will assemble with the mingw assembler), I can do so.

dunno mingw, but it may be able to use "db HEX" like other ASM do.

I don't know how to do that. Rather than try to find out,
I simply tried the mnemonics.

Post by wolfgang kern
33 C9 XOR ecx,ecx
B1 1B MOV CL.0x1B ;make RCXX==0x1B (returns APIC_BASE on AMDs)
0F 32 RDMSR ;may crash if this MSR doesn't exist or CPL>0
0F 09 WBINV ;alters no registers crash if CPL>0.

I started with the easier one, but it didn't assemble.

But this one assembled:

.globl efimain
efimain:

# wbinv

xor ecx,ecx
mov cl, 0x1b
rdmsr

xxx: jmp xxx
#ret

And then executed either the loop or ret that followed, and
both of them perfomed as expected (ie identical to
previous).

Post by ***@gmail.com
But if you're reading the docs anyway, you'll
probably get the answer there.

haven't found anything mentioned about start condition in these docs so
far.

Ok. So reverse-engineering turned out to be easier.

BFN. Paul.

wolfgang kern

2023-03-25 16:02:50 UTC

On 25/03/2023 13:32, ***@gmail.com wrote:
...

Post by ***@gmail.com
Ok, it's still unclear to me what (artificial) constraints
you are placing, and why, but they are clearly genuine
constraints, and that is an interesting use case scenario.

it's easy to explain: I hate folks who sell unfinished stuff.
M$ is the worst on this matter.
So I spent as less money as possible for windoze,
I once bought DOS3.11 and later win98SE, I found copies of XP and win7
and finally I was forced to buy a win10 (I really hate it).
But OTOH it was this stupid M$-stuff which made me create my own OS.

Post by wolfgang kern
So you can do this tests and report what you found.

If you give me a specific instruction to test (ie something
that will assemble with the mingw assembler), I can do so.

dunno mingw, but it may be able to use "db HEX" like other ASM do.

I don't know how to do that. Rather than try to find out,
I simply tried the mnemonics.

I started with the easier one, but it didn't assemble.
.globl efimain
# wbinv
xor ecx,ecx
mov cl, 0x1b
rdmsr
xxx: jmp xxx
#ret

Now you know that the boot code runs in PL0 ?
:):):) LMFAO it would otherwise not make much sense!
...

Post by wolfgang kern
haven't found anything mentioned about start condition in these docs so
far.

Ok. So reverse-engineering turned out to be easier.

and sometimes logical thinking helps too :)
__
wolfgang

muta...@gmail.com

2023-03-26 03:08:47 UTC

Post by ***@gmail.com
Ok, it's still unclear to me what (artificial) constraints
you are placing, and why, but they are clearly genuine
constraints, and that is an interesting use case scenario.

it's easy to explain: I hate folks who sell unfinished stuff.

You can buy a computer and install Freedos and/or Reactos,
neither of which would be "sold" if that's the constraint.

Post by ***@gmail.com
xor ecx,ecx
mov cl, 0x1b
rdmsr
xxx: jmp xxx
#ret

Now you know that the boot code runs in PL0 ?
:):):) LMFAO it would otherwise not make much sense!

It could still make sense - UEFI may have had a requirement
that you need to make an API call, which may or may not be
exitbootservices, to raise the privilege level.

Emperical results show that it is not required for at least
one implementation.

BFN. Paul.

wolfgang kern

2023-03-26 04:48:12 UTC

Post by wolfgang kern
it's easy to explain: I hate folks who sell unfinished stuff.

You can buy a computer and install Freedos and/or Reactos,
neither of which would be "sold" if that's the constraint.

I don't buy computers, only components to assemble PCs.
and why should I install any for me total useless stuff?

after enough info collected I may perhaps start creating any boot.efi by
using either NASM and HEXEDIT or upgrade My HEXTUTOR, all on windoze.
And I could format an image by hand (cumbersome but effective).

and even my current win10 is restricted it might be able to burn CDs
which the machine could boot from.
__
wolfgang

muta...@gmail.com

2023-03-26 05:21:20 UTC

Post by wolfgang kern
it's easy to explain: I hate folks who sell unfinished stuff.

You can buy a computer and install Freedos and/or Reactos,
neither of which would be "sold" if that's the constraint.

I don't buy computers, only components to assemble PCs.
and why should I install any for me total useless stuff?

I don't know why you consider either of the above useless,
when they can avoid the need to hand-format a FAT file
system.

And I think I mentioned before - do you not have a
smartphone? They can format a USB stick too.

Post by wolfgang kern
after enough info collected I may perhaps start creating any boot.efi by
using either NASM and HEXEDIT or upgrade My HEXTUTOR, all on windoze.
And I could format an image by hand (cumbersome but effective).

Do you have a way of writing raw disk sectors to a USB
stick using Windows?

I believe it isn't easy and multiple versions of Win32 Disk
Imager were published for that reason.

BFN. Paul.

wolfgang kern

2023-03-26 05:51:57 UTC

Post by wolfgang kern
it's easy to explain: I hate folks who sell unfinished stuff.

You can buy a computer and install Freedos and/or Reactos,
neither of which would be "sold" if that's the constraint.

I don't buy computers, only components to assemble PCs.
and why should I install any for me total useless stuff?

I don't know why you consider either of the above useless,
when they can avoid the need to hand-format a FAT file
system.

formatting by hand sounds much easier to me.
I owe an USB-FD-drive and I have DOS 6.00 with all tools on FD,
but cannot boot from that and the tools wont work under win10.

Post by ***@gmail.com
And I think I mentioned before - do you not have a
smartphone? They can format a USB stick too.

mine don't have such an option. if this is an app I don't want it.

Do you have a way of writing raw disk sectors to a USB
stick using Windows?

no. windoze may format my sticks (but not FAT32 yet).
I ordered a bunch of newer USB-sticks and will try that again.

Post by ***@gmail.com
I believe it isn't easy and multiple versions of Win32 Disk
Imager were published for that reason.

there is also no IMAGER here.
__
wolfgang

muta...@gmail.com

2023-03-26 06:23:57 UTC

Post by ***@gmail.com
And I think I mentioned before - do you not have a
smartphone? They can format a USB stick too.

mine don't have such an option. if this is an app I don't want it.

It's not an app. It's normally in the settings under
"storage" and then in the external storage down
the bottom.

On any Android phone I've looked at, and I've looked
at quite a lot.

You do need an Android phone that supports OTG
though. Some of my cheaper ones don't. I spent a lot
of effort trying to find the cheapest new Android phone
that supported OTG.

What model smartphones do you have?

Also - do you not have a friend or relative that can format
a FAT-32 USB stick for you?

And apart from that - whenever you buy a USB stick, they
come preformatted as FAT-32.

USB hard drives come preformatted as NTFS in my
experience, but not USB sticks or SD cards. And
there's another option - if you have an SD card from a
camera it is probably already formatted as FAT32,
your PC probably has a SD card reader slot, and your
PC is probably able to boot from SD card too.

BFN. Paul.

muta...@gmail.com

2023-03-26 06:28:49 UTC

Post by ***@gmail.com
USB hard drives come preformatted as NTFS in my
experience, but not USB sticks or SD cards. And
there's another option - if you have an SD card from a
camera it is probably already formatted as FAT32,
your PC probably has a SD card reader slot, and your
PC is probably able to boot from SD card too.

And one more option - if you have a really old Android
phone, or you have some strange modern one that I
bought (but can't identify), it has the ability to present
the user storage as a FAT-32 disk - and may or may not
have an MBR - I think I've seen both - I deliberately made
a change to PDOS when I found this situation of no MBR -
and I think the case where I had an MBR was when I was
using an SD card rather than the internal storage - and
you can connect it to your PC with a cable and boot from it.

BFN. Paul.

wolfgang kern

2023-03-26 07:13:12 UTC

Post by ***@gmail.com
What model smartphones do you have?

Samsung

Post by ***@gmail.com
Also - do you not have a friend or relative that can format
a FAT-32 USB stick for you?
And apart from that - whenever you buy a USB stick, they
come preformatted as FAT-32.

that depends on where you buy it. the 15 pieces I got weren't empty and
they show USB Joliet format. I can access rd/wr files but not format.

nope, no SD reader here.
I have one USB3 HD in use for data backup, I wont mess it.

AND AGAIN, I'm not going to do anything soon right now !
I still gather info in case I once decide to act in future.
__
wolfgang

Dan Cross

2023-03-22 12:19:48 UTC

me too like to see attempts to create an UEFI compliant OS from scratch
w/o reusing C-libraries nor copies from already existing Lonnix/M$.
My own OS retired with me (1985..2018). Contracts end Dec.2024.
I'm too old for a complete restart on new motherboards, they all are
UEFI only now :(

If you have a bootloader that is UEFI compliant, then can it not
load the OS, start it, and then basically get out of the way?

- Dan C.

Scott Lurndal

2023-03-22 14:31:41 UTC

me too like to see attempts to create an UEFI compliant OS from scratch
w/o reusing C-libraries nor copies from already existing Lonnix/M$.
My own OS retired with me (1985..2018). Contracts end Dec.2024.
I'm too old for a complete restart on new motherboards, they all are
UEFI only now :(

If you have a bootloader that is UEFI compliant, then can it not
load the OS, start it, and then basically get out of the way?

That depends. UEFI does more than just load and transfer control,
it isolates the OS from various hardware dependencies making the
OS more portable.

I can't imagine why someone would believe that the legacy BIOS
is in any way superior to UEFI.

Dan Cross

2023-03-22 15:02:44 UTC

me too like to see attempts to create an UEFI compliant OS from scratch
w/o reusing C-libraries nor copies from already existing Lonnix/M$.
My own OS retired with me (1985..2018). Contracts end Dec.2024.
I'm too old for a complete restart on new motherboards, they all are
UEFI only now :(

If you have a bootloader that is UEFI compliant, then can it not
load the OS, start it, and then basically get out of the way?

That depends. UEFI does more than just load and transfer control,
it isolates the OS from various hardware dependencies making the
OS more portable.

Eh....

Post by Scott Lurndal
I can't imagine why someone would believe that the legacy BIOS
is in any way superior to UEFI.

It is certainly not.

However, while it has utility, I wouldn't call it a "good"
interface. Did you catch Mothy Roscoe's OSDI'21 keynote on
rediscovering hardware?
https://www.usenix.org/conference/osdi21/presentation/fri-keynote

Don't get me wrong: interfaces are good. But they're better
when they are well-architected.

UEFI is part of the interface that has congealed.

- Dan C.

wolfgang kern

2023-03-22 16:52:07 UTC

me too like to see attempts to create an UEFI compliant OS from scratch
w/o reusing C-libraries nor copies from already existing Lonnix/M$.
My own OS retired with me (1985..2018). Contracts end Dec.2024.
I'm too old for a complete restart on new motherboards, they all are
UEFI only now :(

If you have a bootloader that is UEFI compliant, then can it not
load the OS, start it, and then basically get out of the way?

this is unfortunately not working. New bought PCs are don't have a
bootloader and my OS don't use GPT. So any try to install failed.
Also failed all attempts to boot from USB, I only see my start-screen
but everything locked up. (would need to rewrite all device drivers).
__
wolfgang

muta...@gmail.com

2023-03-22 22:25:03 UTC

Post by Dan Cross
If you have a bootloader that is UEFI compliant, then can it not
load the OS, start it, and then basically get out of the way?

I have a x64 MBR FAT12 UEFI disk image at pdos.org and it
works fine on all the modern PCs I have tested it on.

I buy really cheap laptops though. Somewhere around US$200.

Some of the laptops are Chromebooks with Tianocore loaded.

Some are Windows laptops, but of course Windows is being
bypassed. All the Windows laptops are running American
Megatrends, which incidentally I couldn't get to do a legacy
boot, even though CSM or similar was shown in the BIOS.

BFN. Paul.

Rick C. Hodgin

2023-03-23 17:32:25 UTC

I have one called "in/OS" for my RISC-like Inspire hardware architecture design.

I'm not posting in this group any more though, so you won't hear about it.

FWIW, I miss this group.

--
Rick C. Hodgin

BGB

2023-03-26 05:44:26 UTC

I have developed an OS of sorts ("TestKern") for a custom CPU ISA (BJX2)
initially mostly because it seemed like too much effort to try to port a
"real" OS (such as Linux or BSD); and ended up going in different
directions on a few fronts.

Some aspects of its design resemble a sort of hybrid of Linux like and
Windows like design choices, with some "twists".

Still pretty rough, crude CLI only, still no GUI or similar (had
experimented with windowed redraw, but its performance fell short of
"usable").

Binary Format:
PEL4, essentially LZ4 compressed PE/COFF;
Loader will also accept regular PE/COFF;
Theoretically, an ELF loader also exists, but is mostly untested.

The ABI does tweak some things slightly, as the PE/COFF image is
interpreted as being two major parts:
A read-only area containing ".text" and other read-only sections;
A read/write are containing ".data" and ".bss" and similar;
These may be put into different areas in the address space;
Depends mostly on the discretion of the loader;
The kernel image is loaded as a single monolithic part.

Some other sections and structures differ from those in "mainline"
PE/COFF variants. Similarly, PEL4 binaries omit the MZ stub and header
(and start with a PEL4 magic followed directly by the COFF headers; with
the headers uncompressed but all the section contents are LZ4 compressed).

I tested a few compression schemes, but LZ4 seemed to work out as best
"on average" in this use-case (I have another compression scheme which
works better on average for general data; but tended to do worse than
LZ4 with executable code).

Main reason binaries are LZ compressed is mostly to make them load
faster from an SDcard. Similarly, no Deflate both for complexity reasons
and because Deflate would be slower to decompress than the speed of the
SDcard.

Address Space:
Monolithic 48 (flat) or 96 bit (segmented / "quadrants");
In the 96 bit mode, only the low 48b is linear;
Another 48 bits functions as the quadrant address.

In 96-bit mode, each program can be given its own local 48-bit quadrant
while still being part of a larger shared 96 bit space, and it is
possible to use 128-bit pointers to address anywhere inside this space.

However, the default pointer size remains as 64-bits with 48-bits for
the address and 16-bit for tag metadata (usually zeroed for bare
pointers in C, or non-zero for bounds-checked pointers; also usable for
dynamic type-tag handling).

When using 64-bit pointers, it is functionally similar to memory
addressing in the 65C816 (or segmented addressing in x86, though this is
a less accurate comparison). Mostly this is done because using 128-bit
pointers as the default would be overkill (in C, the pointers use a
"__huge" modifier, Say: "void * __huge ptr;").

So, with 64-bit pointers, one can access up to 256TB, or 64 RB/RiB
(ronnabyte/robibyte, with the newer SI prefixes), or 256 TQu
("teraquad", if one assumes each "quad" is 256TB).

In 48-bit mode, memory management is via a 3-level page-table (with a
16K page-size and 64-bit page-table entries).

In 96-bit mode, the page-tables are mostly replaced by B-Trees or
"hybrid trees" where the high-order levels are managed by a B-Tree, but
the last 1 or 2 levels are page-tables.

Partly this is because 3-level page tables are reasonably efficient for
a 48-bit space, but quickly become very inefficient with a large
sparsely filled address space (with an 8-level page-table, most upper
level pages are almost entirely empty).

Also, by various metrics, the B-Tree won out against its nearest
competitors, namely an AVL Tree and a Hash-Table Tree. A hash-table can
potentially be faster in simple cases, O(1) best-case vs O(log2 n), in
experimental tests the B-Tree retained better "average case" performance
(as the node got closer to full hash-table speed falls in the toilet).
In terms of both speed and memory use, the B-Tree beat out the AVL Tree.

Memory Protection:
Still not well developed in practice, but in theory, there is ACL
checking on memory pages (so, for example, a task PID or similar be can
be checked against an ACL to determine whether it has been granted a
particular access to a particular ACLID), with the CPU throwing an
access-fault if the program tries to access memory that it has not been
granted access. Each memory page may be assigned to an ACL.

This partly extends an older system known as VUGID which applied
(non-ACL) User/Group/Other checks to pages (using a similar model to
that used in the Unix/Linux filesystems). Conceptually, an ACL is more
general (and has some similarities to file protection in Windows / NTFS).

A lot of this is partially built into the MMU; Though, in my case I am
using a software-managed TLB and ACLB (some of this would likely be
impractical if all of the memory management were done entirely in hardware).

Note that despite some overlap (and both systems having support for
bounds-checking pointers), the abstract memory model differs
significantly from something like CHERI. However, I personally suspect
that my design is "more practical" as it doesn't require using larger
pointers or mess up traditional C coding practices (the constraints
imposed by CHERI are, within the limits of the letter of the C standard;
a notable departure from the "common expectations" of how C code works
on typical flat-addressed machines).

While one could argue similar about the 96b address space, likely 99.9%
of programs can probably ignore that anything exists outside the low 48
bits (it's local address space just floating at some random location
within an endless black sea).

Filesystem:
Mostly FAT32 for now, mostly as Windows interferes with me using
anything. Had experimented with hacking a few Unix-like features onto
FAT using a vaguely similar approach to UMSDOS albeit still using LFNs
(the internal-use SFN is keyed to any additional file metadata though).

HAL API:
I have an API known as TKGDI which provides some interfaces for hardware
interfacing. It sort of resembles a hybrid of Windows GDI, MMSYS, and
VFW. It can also export an OpenGL interface.

It provides a C style API, but internally works using a mechanism
similar to COM objects, where the method calls to the object may be
routed across the system-call interface, potentially to objects located
in a different process.

Shared data buffers can be "imported"/"exported" and turned into a
handle used to access the object from another logical address space.

Some other data is passed through the dynamic type-system as it is
easier to wrangle dynamically typed data between processes than
static-typed data (I didn't want to need to implement an IDL/Schema
system; and it seemed "less bad" to require data to be passed using a
dynamic type system when potentially crossing between logical address
spaces; even if both likely exist within the same larger 96-bit space).

CPU ISA (other aspects):

Registers:
64 GPRs, each 64-bits;
May be paired into 32 logical 128-bit registers;
FPU and SIMD also use the GPR space.
1x Binary64 (64b)
2x Binary32 (64b)
4x Binary16 (64b)
2x Binary64 (128b)
4x Binary32 (128b)
Load/Store Architecture.
Typical of many RISCs.

Execution Type: VLIW / Explicitly Parallel:
Instructions are put into "bundles" and executed in parallel;
Up to 3 instructions per clock-cycle.
Compiler is mostly responsible for instruction scheduling;
Instructions may also be predicated;
Either executed or skipped depending on a condition and flag bit.

Design partly evolved from the Hitachi SuperH ISA, with some influences
from TMS320 and RISC-V, despite its high-level aspects having some
features in common with Itanium / IA-64. In some ways, its bundling
semantics have more in common with IA-64 than with TMS320.

Unlike IA-64, it's code density isn't total crap.

For scalar code, instructions are variable-length 16/32/64/96 bit.
For VLIW code, instructions are 32/64/96 bits;
Though, only 32-bit forms may be used in bundles.

The 64/96 bit encodings are essentially instruction bundles functioning
as a single logical instruction (these and 128-bit SIMD instructions may
only execute one at a time).

Peak theoretical SIMD performance being roughly 200 MFLOP/s at 50MHz (on
an FPGA).

Some examples of 32-bit instructions:
MOV Imm16u, Rn // Rn = Imm16u
MOV.Q (Rm, Disp9u), Rn // 64-bit memory load (4 kB)
MOV.Q (Rm, Ro), Rn // 64-bit memory load (indexed)
ADD Rm, Imm9u, Rn // Rn = Rm + Imm9u
MULS.L Rm, Ro, Rn // Rn = Rm * Ro
PMULX.F Rm, Ro, Rn // Rn = Rm * Ro (SIMD 4x Binary32)
...

Some examples of 64-bit instructions:
MOV.Q (Rm, Disp33s), Rn //64-bit memory load (32GB)
ADD Rm, Imm33s, Rn // Rn = Rm + Imm33s
...

Some examples of 96-bit instructions:
MOV Imm64, Rn //Rn = Imm64
ADD Imm64, Rn //Rn = Rn + Imm64
PMULX.F Xm, Imm56fv, Xn //Xn = Xm * vec4(x, y, z, w)
...

Though, the ability to encode vector immediates into SIMD ops (as well
as SIMD-ops with built-in shuffle) is a "bleeding edge" (still not fully
implemented) feature. These are mostly intended for a few niche cases
involving intensive SIMD ops (cough... neural nets...).

...

Example 3-register 32-bit instruction format (two 16-bit words):
111P-ZwZZ-nnnn-mmmm ZZZZ-qnmo-oooo-ZZZZ (Baseline, R0..R31)
0111-wnmo-nnnn-mmmm ZZZZ-qnmo-oooo-ZZZZ (XGPR, R0..R63)
NMOP-ZwZZ-nnnn-mmmm ZZZZ-qnmo-oooo-ZZZZ (XG2, R0..R63)

Example 3-register-Imm9 32-bit instruction format (two 16-bit words):
111P-ZwZZ-nnnn-mmmm ZZZZ-qnmi-iiii-iiii (Baseline, R0..R31)
1001-wnmZ-nnnn-mmmm ZZZZ-qnmi-iiii-iiii (XGPR, R0..R63)
NMIP-ZwZZ-nnnn-mmmm ZZZZ-qnmi-iiii-iiii (XG2, R0..R63)

Whrere:
Z=Opcode
q=Data Size or Z
n=Rn, m=Rm, o=Ro
N/M/O: R32..R63 (Inverted)
P=Predicated (Inverted), Unconditional if P=1.
w=Wide-Execute (P=1), True/False (P=0)
i=Immediate
I=Immediate or Sign Extension (Inverted)
Imm9u -> Imm10u
Disp9u -> Disp10s

Baseline encodings are limited to R0..R31.
XGPR allows R32..R63 but for a subset of the ISA.

XG2 Mode is an alternate ISA mode which loses the ability to encode
16-bit instructions, but gains the ability for the entire ISA to use all
64 GPRs in a more orthogonal way.

There is also a (not well tested) RV64I mode, where the CPU can execute
RISC-V RV64IM code (mostly userland only), noting as my CPU core's
pipeline was mostly a superset of what was required for RISC-V (in lower
end variants).

The CPU ISA, OS, and my C compiler, have all been sort of tied together
thus far...

Other funkiness:

The display interface is cell-oriented rather than a raster-oriented
framebuffer; and it has a combination of text-mode and color-cell mode
(along vaguely similar lines to a NES).

Modes:
40x25, 80x25, and 80x50 cell.
320x200, 640x200, and 640x400 is used for color-cell graphics.
320x200 RGB555 / hi-color
320x200 or 640x400 256-color fixed palette
640x400 16-color
800x600 4-color
Multiple palettes available.
1024x768 monochrome.
Multiple colors available.

The 4-color and monochrome modes have a few Bayer-Array like palettes,
which allow crudely approximating full color graphics.

The 256 color mode currently has a hard-wired palette:
00..7F: 0rrgggbb //RGB232
80..FF: 1iiiirgb //16 levels of 8 primary colors

Partly this is because on the FPGA boards I am using, I don't really
have enough memory bandwidth to manage something like 640x480 hi-color
or similar.

Also has a PCM audio output, as a small looping PCM buffer (8-bit linear
PCM or A-law), and an FM synthesis module (for MIDI playback), which
follows an OPL-like design but (sadly) fails to recreate the original
Adlib/SoundBlaster sound for MIDI playback.

Had been recently working (without so much success thus far) at trying
to get a USB Host interface working.

Well, among lots of other funkiness.

...

Post by Dan Cross
- Dan C.

James Harris

2023-12-12 21:15:22 UTC

I haven't read the whole thread so someone else may have already made
this point but your reference to 'modern hardware' seems to be a curious
one. Many of the goals of an OS, AISI, involve providing an abstraction
which does two things:

1. provides a common view of hardware so that programs running under the
OS don't have to care about hardware specifics,

2. allows exploitation of hardware only in so far as not to conflict
with point 1.

Many of the past discussions here have been about coping with hardware
differences.

Am curious, though, if you are still reading what novel modern hardware
facilities did you have in mind? (Presumably significantly more than
paging and privilege levels, etc.)

--
James Harris

Dan Cross

2023-12-13 15:24:50 UTC