Post by Paul EdwardsPost by BGBBut, now I am an aging millennial and have arguably not accomplished all
that much with my life.
Didn't you email me decades ago to get some changes implemented
to PDPCLIB and you mentioned you were writing a phenomenal
number of lines of code per day? Where did all that effort go?
FWIW:
I ended up with a 3D engine, which was around a 1 MLOC, sort of like
Minecraft with a Doom3 style renderer. No one cared, performance wasn't
so good (was painfully laggy), and this project fizzled.
Part of the poor performance was the use of a conservative garbage
collector, and rampant memory leaks, ... Another part was was "Minecraft
style terrain rendering and stencil shadows don't mix well". Though, for
small light sources, could subset the scene geometry mostly to a
bounding-box around the light source.
But, the sun, well, the sun was kinda evil. Did later move to
shadow-maps for the sun though (though, IIRC, did RGB shadow maps to
allow for colored shadows through colored glass).
Then I wrote a new 3D engine ground-up, which was smaller and had better
performance. Few people cared, I lost motivation, and eventually it
fizzled as well. Was roughly around 0.5 MLOC, IIRC.
It had replaced the complex dynamic lighting with the use of
vertex-color lighting (with a single big rendering pass).
I started on my CPU ISA project, which checking, is around 2 MLOC (for
the C parts.
It is ~ 3.8 MLOC total, if one includes a lot of ASM and C++ code; but a
fair chunk of this is auto-generated (Verilator output, or debug ASM
output from my compiler).
There is also around 0.8 MLOC of Verilog in my project; but this drops
to 200 kLOC if only counting the current CPU core.
Ironically, the OS for my current ISA project has reused some parts from
my past 3D engine projects.
In the course of all this, ended up doing roughly 3 separate
re-implementations of the OpenGL API (the 3rd version was written to try
to leverage special features of my ISA; though was originally written to
assume a plain software renderer, and since implementing a ).
In my current project, I have ports of GLQuake and Quake 3 Arena working
on it; though performance isn't good on a 50MHz CPU.
Ironically, parts of PDPCLIB still remain as a core part of the "OS",
though I had ended up rewriting a fair chunk of it to better fit my
use-case (the "string.c" and "math.c" stuff ended up almost entirely
rewritten, though a fair chunk of "stdio.c" and similar remains intact).
It was also expanded out to cover much of C99 and parts of C11 and C23.
Some wonky modifications were made to support DLLs, which ended up
working in an unusual way in my case:
The main binary essentially exports a COM interface to its C library;
Most of the loaded DLLs have ended up importing this COM interface,
which provides things like malloc/free, stdio backend stuff, ...
It also has a small makeshift GUI, though mostly just displays a shell
window that can be used to launch programs.
Besides my own ISA, my CPU core also runs RISC-V.
There is a possible TODO effort of trying to implement the Linux syscall
interface for RISC-V Mode, which could potentially allow me to run
binaries letting GCC use the "native" GLIBC, which could make porting
software to it easier (vs the hassle of getting GCC to use my own
runtime libraries; or trying to get programs to build using my own
compiler as a cross-compiler).
Though, I did more or less get my compiler to pretend to be GCC well
enough that for small programs, it is possible to trick "./configure"
scripts to use it as a cross compiler (doesn't scale very well, as apart
from some core POSIX libraries, most anything else is absent).
Where, for my own ISA, I am using BGBCC.
BGBCC is ~ 250 kLOC, and mostly compiles C;
Also compiles BGBScript, which sorta resembles ActionScript;
And, BGBScript2, which sorta resembles Java mixed with C#;
Albeit, unlike Java and C#, it uses manual and zone allocation.
Technically could be mixed with C, all using the same ABI;
Also an EC++ like subset of C++.
But, kinda moot as no "Modern C++" stuff has any hope of working.
But, for my current uses, C is dominant.
It is sorta wonky in that it does not use traditional object files.
It compiles into a stack-oriented bytecode and "links" from this.
The bytecode IR could be loosely compared with MSIL / CIL.
ASM code is preprocessed and forwarded as text blobs.
The backend then produces the final PE/COFF images.
Though, this mutated some as well:
Lacks MZ stub / header;
PE image is typically LZ4 compressed.
LZ4 compression makes the loading process faster.
Resource section was replaced with a WAD2 variant.
Made more sense to me than the original PE/COFF resource section.
Compiler also has a built-in format converter.
Say, to convert TGA or PNG into BMP (*1), ...
*1: General resource-section formats:
Graphics:
BMP, 4/8/16/24/32 bit.
Ye Olde standard BMP.
For 16 and 256 color, fixed palettes are used.
BMPA, 4/8 bit with a transparent color.
Basically standard, but with a transparent color.
Generally, the High-Intensity Magenta is transparent.
Or, #FF55FF (or, Color 13 in the 16-color palette)
BMP+CRAM: 2 bpp 256-color, image encoded as 8-bit CRAM.
Supports transparency in a limited form:
Only 1 non-transparent color per 4x4 block,
vs 2 colors for opaque blocks.
QOI: An image in the QOI format (lossless)
LCIF: Resembles a QOI/CRAM hybrid, lossy low/intermediate quality.
Though, BMP+CRAM is faster and has lower overhead.
UPIC: Resembles a Rice-coded JPEG
Optimized for a small low-memory-overhead decoder.
Lossy or Lossless, higher quality, but comparably slow.
Audio:
WAV, mostly PCM, A-Law, or ADPCM.
BGBCC originally started as a fork off of my BGBScript VM, which was
used as the main scripting language in my first 3D engine.
By the 2nd 3D engine, it had partly been replaced by a VM running my
(then) newer BGBScript2 language, with the engine written as a mix of C
and BGBScript2.
While I could technically use BGBScript2 in my TestKern OS, it is almost
entirely C, only really using BGBScript2 for some small test cases (it
is technically possible to use both BS and BS2 in kernel and bare-metal
contexts; and there is partial ISA level assistance for things like
tagged pointers and dynamic type-checking). Where, BS2 retains (from BS,
and its JS/AS ancestors) the ability to use optional dynamic types and
ex-nihilo objects (also BGBCC technically allows doing so in C as well,
with some non-standard syntax, but doing so is "kinda cursed").
Ironically, I am using a memory protection scheme in my ISA based on
performing ACL checks on memory pages. The basic idea for this scheme
was carried over from my original BGBScript VM (where it was applied per
object), where the idea for the scheme was (ironically) inspired partly
by how object security was passed off in the "Tron 2.0" game (in
context, as a more convoluted way of passing off the use of keycards for
doors). But, I was left thinking at the time that the idea actually
sorta made sense. But, in its present form mostly involves applying
filesystem-style checks to pages (the MMU remembers this, but raises an
exception whenever it needs the OS to sort out whether a given key can
access a given ACL).
Well, and also the use of pointers in my ISA with a 48-bit address, and
16 bits of tag metadata in the high order bits, was also itself partly a
carry-over from my Script VMs.
Near the end of my 2nd 3D engine (before the project fizzled out
entirely): It also gained the ability to load BJX2 images into the 3D
engine. In the effect, the 3D engine would itself take on a role like an
OS, effectively running logical processes inside the VM (though, there
wasn't really an API to glue these into the game world).
IIRC, the idea I think was to make the "game server" able to run
programs OS style, which could then run parts of the game logic (rather
than necessarily using my BGBSCript2 language running in a VM;
potentially the BS2 VM code could be ported to BGBCC and run inside the
BJX2 VM). Though, potentially, one could also make a case for using
RISC-V ELF images. Wouldn't necessarily want to run native x86-64 code
as it would be desirable to be able to sandbox the programs inside of a
VM. In such a case, the idea would be that things like voxel
manipulation or interaction with world entities could be via COM
objects, or potentially game objects could signal events into the script
programs.
Can note that my BJX2 project was preceded by BJX1, where BJX1 started
out as a modified version of the Hitachi SH-4 ISA (most popularly used
in the SEGA Dreamcast). I had revived BGBCC initially as I needed a
compiler to target BJX1 (and SH-4). As BJX1 turned into a horrible mess
(turned 64-bit, and fragmented into multiple variants), I eventually did
a "partial reboot".
At the ASM level, initially BJX2 was very similar to BJX1, mostly
carrying over the same ASM and ABI, but with minor changes (and gaining
some features and notation inspired by the TMS320). The BJX2 ISA mutated
over time, and has since also fragmented to some extent (and its current
form also has some similarities to SH-5).
It has since drifted towards being more like RISC-V in some areas,
mostly because my CPU core can now also run RISC-V code (and, if RISC-V
needs a feature, and it is useful, may as well also have it in my own ISA).
ASM syntax/style mostly borrowed from SH-4, which seems to be in a
similar category that also includes the likes of MSP430, M68K, PDP-11,
and VAX. Well, as opposed to RISC-V using a more MIPS-like style.
I also don't really have a "proper" userland as of yet, more the kernel,
shell, and most basic programs, all exist as a single binary (so, say,
if you type "ls", the shell handles it itself; with shell instances as
kernel mode threads).
Any "actual" programs are loaded and then spawned as a new process.
Only recently-ish added the ability to redirect IO, but still doesn't
support piping IO between programs.
Supports basic shell-scripts, but lacks most more advanced shell
features (non-trivial Bash scripts will not work).
There was a 3rd 3D engine of mine, mostly because my 2nd 3D engine would
have still been too heavyweight to run on my CPU core (tried to write
something Minecraft-like that would run in a similar memory footprint to
Quake and was fast enough to be tolerable on a 50MHz CPU).
Between the engines:
Chunk Size: 16x16x16 in both engines;
Region Size: 16x16x16 in 2nd engine, 8x8x8 in 3rd.
32x32x8 in first engine.
Thus, in 3rd engine, each region was a 128x128x128 meter cube.
Block Storage:
1st engine: 8 bit index or unpacked;
2nd engine: 4/8/12 bit index into table of blocks;
3rd engine: 4/8 bit index into block table, or unpacked block array.
World Size:
1st engine: Planar
2nd engine: 1024km (1048576 meters), world wraps on edge
3rd engine: 64km (65536 meters), world wraps on edge
Rendering:
1st engine: global vertex arrays, filled from each chunk
2nd engine: Per-chunk vertex arrays
3rd engine: Raycast, visible blocks drawn into global vertex arrays.
Chunk Storage:
1st engine: RLEW (same format as used for maps in Wolf3D and ROTT)
2nd engine: LZ77 + AdRiceSTF
3rd engine: RP2 (similar to LZ4)
Graphics storage:
1st engine: JPEG (modified to support Alpha channel)
2nd engine: BMP + BTIC4B (8x8 Color-Cell, AdRice Bitstream)
3rd engine: DDS (DXT1)
VFS File Storage:
1st engine: ZIP
2nd engine: BTPAK (Hierarchical Central Directory, Deflate)
Large files broken up into 1MB fragments;
3rd engine: WAD4 (Hierarchical Central Directory, RP2)
Large files broken into 128K fragments.
Audio:
1st engine: WAV (PCM)
2nd & 3rd engine: WAV (IMA ADPCM)
Both 2nd and 3rd engine used the same block-type numbers and the same
texture atlas.
Both 2nd and 3rd engine had used mostly sprite graphics, for my first 3D
engine, I had used 3D models (and skeletal animation), but this was a
lot of effort.
I then noted that sprite graphics still worked well in Doom, and
attempted to mimic the use of sprites, though generally using 4 angles
rather than 8, as 4 was easier to draw. Also using a trick as seen in
some old RPG's where one could pull off idle animations and walking by
horizontally flipping the sprite on a timer.
Initial goal (before the 2nd engine effort fizzled out) was to try to
build something like Undertale, but this was more effort, and I was
lacking a good system for dialog and managing game-event dependency trees.
My 3rd engine never got much past "try to make something that works on
my ISA and can fit in under 40-60MB of RAM".
One minor difference was for live entity serialization within regions,
where my 2nd engine had mostly embedded data within ASCII strings,
whereas the 3rd engine had used binary-serialized XML blobs (reusing the
XML code from BGBCC, where for better or worse; BGBCC had uses XML DOM
style ASTs; but reworked to be a lot more efficient than the original DOM).
Also, some amount of specialized image and video codecs, etc.
There is an experimental video player and Mod/S3M player, though not at
present generalized enough to be usable as media players (would need
some level of UI for this; thus far these load a hard-coded file and
just play it in a loop).
And, some on/off fiddling with things like Neural Nets, etc.
Recently I wrote a tool to import UFO / GLIF fonts and convert them to
an custom font format (mostly as the actual TTF format seemed needlessly
complicated). This was along with the code to render this style of font.
Unclear if it will replace the use of bitmap fonts and SDFs.
Where:
Bitmap font:
Specialized for specific glyph sizes;
Looks good at that size;
Don't really scale.
SDF font:
Scalable (works best for medium glyphs);
Relatively cheap in a computational sense;
But, relatively bulky and eat a lot of memory.
I stored them mostly as 8bpp BMP images, 4-bit X/Y.
Where, each 16x16 glyph page is a 256256 BMP,
Variable / Geometric Font:
Scalable (but works best for large glyphs);
Attempts to draw small glyphs give poor results ATM.
Currently need to draw at 4x final size then downsample.
Higher per-pixel cost;
Less memory needed to hold font;
Can be used to generate SDF's or triangles.
For my ISA / OS project, the fonts and some other things had been
carried over from my 3D engine projects. Well, along with a lot of the
VFS and memory management code (wasn't too much effort to adapt my 3D
engine VFS code to work as an OS VFS).
Main practical difference being that, for an OS VFS, it has a FAT32
driver and similar.
But, I have slowed down in recent years I suspect.
Post by Paul EdwardsRegardless, what sort of thing would you consider to be
"accomplished a significant amount"? You're not going to
single-handedly reproduce Windows 11. So if that is the
bar, no-one at all has accomplished much. It's even difficult
to credit Windows itself. Who are you going to credit?
Tim Paterson? Or Bill Gates's father's (or was it his mother's?)
money?
Note that I am not dismissing Bill Gates's technical achievements
with Microsoft BASIC, but that's not Windows 11 by a very
very very long shot.
Dunno...
Just is seems like a lot of other people are getting lots of
recognition, seem to be doing well off financially, etc.
Meanwhile, I just sort of end up poking at stuff, and implementing
stuff, and it seems like regardless of what I do, no one gives a crap,
or like I am little better off than had I done nothing at all...