news galore

[[e-mail]]

I have been struggling with when to use XON, but just
now I realized that I don't need XON to communicate
with eternal-september. The NNTP protocol already has
appropriate markers so that I know when to switch
to/from reading/writing.

So I now have a design for a C90-compliant news reader.

It will open a device, such as "COM1" on PDOS/386 or
"0x10001" on z/PDOS, as "r+b" and start reading characters
until it hits whatever endless-september uses as end-of-line
marker (they are the server, they can do whatever the fuck
they want - I'm pretty sure it is CRLF), and translates from
ASCII to the local character set.

This will get the 0200 line that eternal-september puts out.

There will be an fseek SEEK_CUR 0 as required by the C90
standard prior to then doing a write, which needs to be sent
in ASCII as "LIST\r\n" (that's what I currently do, but it isn't
actually necessary).

This setup will replace the crappy TCP/IP calls that I
currently do.

BFN. Paul.

[Scott Lurndal]

Post by ***@gmail.com
It will open a device, such as "COM1" on PDOS/386 or
"0x10001" on z/PDOS, as "r+b" and start reading characters
until it hits whatever endless-september uses as end-of-line
marker (they are the server, they can do whatever the fuck
they want - I'm pretty sure it is CRLF), and translates from
ASCII to the local character set.

You might want to read through this first.

https://datatracker.ietf.org/doc/html/rfc3977

"The character set for all NNTP commands is UTF-8 [RFC3629]. Commands
in NNTP MUST consist of a keyword, which MAY be followed by one or
more arguments. A CRLF pair MUST terminate all commands. Multiple
commands MUST NOT be on the same line. Unless otherwise noted
elsewhere in this document, arguments SHOULD consist of printable US-
ASCII characters. Keywords and arguments MUST each be separated by
one or more space or TAB characters. Command lines MUST NOT exceed
512 octets, which includes the terminating CRLF pair. The arguments
MUST NOT exceed 497 octets. A server MAY relax these limits for
commands defined in an extension."

Post by ***@gmail.com
This will get the 0200 line that eternal-september puts out.

The first response you'll get from most NNTP servers
will be a request for authentication after you send
MODE READER. That will be a response code of "480"
(see appendix C).

if (ec == 480) {
/*
* authentication required
*/
snprintf(buffer, sizeof(buffer), "authinfo user %s\r\n",
conp->username);
fputs(buffer, conp->writer);

bp = fgets(buffer, sizeof(buffer), conp->reader);
if (buffer[0] != '3') {
fprintf(stderr, "auth user returns %s\n", buffer);
}
snprintf(buffer, sizeof(buffer), "authinfo pass %s\r\n",
conp->password);
fputs(buffer, conp->writer);
bp = fgets(buffer, sizeof(buffer), conp->reader);
if (buffer[0] != '2') {
fprintf(stderr, "auth pass returns %s\n", buffer);
}

goto retry;
}

Post by ***@gmail.com
There will be an fseek SEEK_CUR 0 as required by the C90
standard prior to then doing a write,

Where, exactly, does C90 require an fseek on a non-seekable
file like a UART?

[[e-mail]]

"The character set for all NNTP commands is UTF-8 [RFC3629]. Commands
in NNTP MUST consist of a keyword, which MAY be followed by one or
more arguments. A CRLF pair MUST terminate all commands.

I'll be doing all that, given that I'm only interested in
ASCII messages from eternal-september.

Post by ***@gmail.com
This will get the 0200 line that eternal-september puts out.

The first response you'll get from most NNTP servers
will be a request for authentication after you send
MODE READER. That will be a response code of "480"
(see appendix C).

Ok, the important thing is that the protocol lets me know
when I need to switch to/from reading/writing. Because
I want to write C90 code.

Post by ***@gmail.com
There will be an fseek SEEK_CUR 0 as required by the C90
standard prior to then doing a write,

Where, exactly, does C90 require an fseek on a non-seekable
file like a UART?

7.9.5.3

When a file is opened with update mode ('+' as the second or third character in the above
Iist of mode argument values), both input and output may be performed on the associated stream.
However, output may not be directly followed by input without an intervening call to the
fflush function or to a file positioning function (fseek, fsetpos, or rewind), and input
may not be directly followed by output without an intervening call to a file positioning function,
unless the input operation encounters end-of-file.


There is no exception for "unless the stream is associated
with a UART, of course!".

BFN. Paul.

[Scott Lurndal]

Post by ***@gmail.com

Post by Scott Lurndal

Post by ***@gmail.com
There will be an fseek SEEK_CUR 0 as required by the C90
standard prior to then doing a write,

Where, exactly, does C90 require an fseek on a non-seekable
file like a UART?

7.9.5.3
When a file is opened with update mode ('+' as the second or third character in the above
Iist of mode argument values), both input and output may be performed on the associated stream.
However, output may not be directly followed by input without an intervening call to the
fflush function or to a file positioning function (fseek, fsetpos, or rewind), and input
may not be directly followed by output without an intervening call to a file positioning function,
unless the input operation encounters end-of-file.
There is no exception for "unless the stream is associated
with a UART, of course!".

You may want to go back and re-read the first paragraph of the section entitled "Files":

"If a file can support positioning requests (such as a
disk file, as opposed to a terminal), then a file position
indicator associated with the stream is positioned at the start
(character number zero) of the file, unless the file is
opened with append mode in which case it is implementation-defined
whether the file position indicator is initially positioned at the
beginning or the end of the file. The file position indicator is
maintained by subsequent reads, writes, and positioning requests, to
facilitate an orderly progression through the file."

In all cases, fseek can return an implementation defined error code,

"If successful, the ftell function returns the current value of the file position indicator
for the stream. On failure, the ftell function returns -1L and stores an
implementation-defined positive value in errno."

(fseek refers to ftell for error conditions)

For usable real-world operating systems, fseek will fail (return -1L) and
set errno to EBADF for attempts to seek on a non-seekable stream (like
that associated with a terminal or TCP connection).

[[e-mail]]

Post by Scott Lurndal
"If a file can support positioning requests (such as a

The file stream associated with COM1 will only support
positioning requests of offset 0 from the current position,
or in the future, maybe positive offsets (read and discard
bytes from the UART).

Post by Scott Lurndal
For usable real-world operating systems, fseek will fail (return -1L) and
set errno to EBADF for attempts to seek on a non-seekable stream (like
that associated with a terminal or TCP connection).

My intention for PDOS/386 and z/PDOS is not to copy the
bad design decisions of others, but to see what can be done
within the confines of C90. And it would appear that that
includes being able to write a news reader.

BTW, another thing I will need to do (I think) is to setvbuf
the COM1 stream to make it unbuffered.

BFN. Paul.

[Scott Lurndal]

Post by ***@gmail.com

Post by Scott Lurndal
"If a file can support positioning requests (such as a

The file stream associated with COM1 will only support
positioning requests of offset 0 from the current position,
or in the future, maybe positive offsets (read and discard
bytes from the UART).

Post by Scott Lurndal
For usable real-world operating systems, fseek will fail (return -1L) and
set errno to EBADF for attempts to seek on a non-seekable stream (like
that associated with a terminal or TCP connection).

My intention for PDOS/386 and z/PDOS is not to copy the
bad design decisions of others, but to see what can be done
within the confines of C90. And it would appear that that
includes being able to write a news reader.
BTW, another thing I will need to do (I think) is to setvbuf
the COM1 stream to make it unbuffered.

The NNTP protocol is line oriented, so a line-buffered stream
is just fine.

[[e-mail]]

Post by Scott Lurndal

Post by ***@gmail.com
BTW, another thing I will need to do (I think) is to setvbuf
the COM1 stream to make it unbuffered.

The NNTP protocol is line oriented, so a line-buffered stream
is just fine.

Even assuming ASCII, and even assuming LF endings,
what is the interaction here? Ultimately the C library is
going to do some sort of read() call on the UART/TCPIP
stream. It needs to know to return immediately, not wait
for a fixed-length buffer to be filled.

How can that be done other than insisting on a read of
length 1, and how can that be done other than switching
off buffering with setvbuf?

Thanks. Paul.

[Scott Lurndal]

Post by ***@gmail.com

Post by Scott Lurndal

Post by ***@gmail.com
BTW, another thing I will need to do (I think) is to setvbuf
the COM1 stream to make it unbuffered.

The NNTP protocol is line oriented, so a line-buffered stream
is just fine.

Even assuming ASCII, and even assuming LF endings,
what is the interaction here? Ultimately the C library is
going to do some sort of read() call on the UART/TCPIP
stream. It needs to know to return immediately, not wait
for a fixed-length buffer to be filled.
How can that be done other than insisting on a read of
length 1, and how can that be done other than switching
off buffering with setvbuf?

I posted a C code fragment from an existing NNTP client
that uses 'fgets' and 'fputs' on a line-oriented stream
(associated with a TCP connection). The C library handles
it just fine. There are a half-dozen or more open source
C libraries around that you can download and see how they
do it.

cp->sd = socket(AF_INET, SOCK_STREAM, 0);
if (cp->sd == -1) {
fprintf(stderr, "%s: Unable to create socket: %s\n",
myname, strerror(errno));
return NULL;
}

fprintf(stderr, "Connecting to %s using port %d\n",
cp->hostname, port);
addr.sin_family = hp->h_addrtype;
addr.sin_port = port;
fprintf(stderr, "IP address is %u.%u.%u.%u\n",
(unsigned char)hp->h_addr[0], (unsigned char)hp->h_addr[1],
(unsigned char)hp->h_addr[2], (unsigned char)hp->h_addr[3]);
memcpy((char *)&addr.sin_addr, *hp->h_addr_list, hp->h_length);

diag = connect(cp->sd, (struct sockaddr *)&addr, sizeof(addr));
if (diag == -1) {
fprintf(stderr, "%s: Unable to establish connection: %s\n",
myname, strerror(errno));
close(cp->sd);
return NULL;
}

cp->reader = fdopen(cp->sd, "rb");
if (cp->reader == NULL){
fprintf(stderr, "%s: Unable to reopen file: %s\n",
myname, strerror(errno));
close(cp->sd);
return NULL;
}

cp->writer = fdopen(cp->sd, "wb");
if (cp->writer == NULL){
fprintf(stderr, "%s: Unable to reopen file: %s\n",
myname, strerror(errno));
close(cp->sd);
return NULL;
}
setvbuf(cp->writer, (char *)NULL, _IOLBF, 0);

[[e-mail]]

Post by Scott Lurndal
I posted a C code fragment from an existing NNTP client
that uses 'fgets' and 'fputs' on a line-oriented stream
(associated with a TCP connection). The C library handles
it just fine. There are a half-dozen or more open source
C libraries around that you can download and see how they
do it.

I thought of how it could be done - the fact that the stream
is line buffered would cause the C library to do reads of
1 byte.

Only if a stream is fully buffered would it do a fixed size read.

But this leads to a different problem - the C90 standard says:

When opened, a stream is fully buffered if and only if it can be
determined not to refer to an interactive device.

That sounds like it has Unix origins. I'm not sure that Windows
and MVS provide calls to neatly determine whether a device can
be fully buffered or not. I'm not sure that that is necessarily known
even at the hardware level.

This is a philosophy question. I'm not just after someone else's
implementation. I'd like to derive the philosophy from first
principles, and I'm willing to change the C90 standard if it is
insufficient.

I'm wondering whether an application should be the one to
signal it is wanting to do interaction at the byte or line level
rather than being left to the C library to figure it out for itself.
With files being fully-buffered by default, which is the normal
case for disk files.

I'm talking about files other than stdin/stdout/stderr which I
am happy to have as line-buffered by default.

BFN. Paul.

[Branimir Maksimovic]

Post by Scott Lurndal

Post by ***@gmail.com

Post by Scott Lurndal

Post by ***@gmail.com
BTW, another thing I will need to do (I think) is to setvbuf
the COM1 stream to make it unbuffered.

The NNTP protocol is line oriented, so a line-buffered stream
is just fine.

Even assuming ASCII, and even assuming LF endings,
what is the interaction here? Ultimately the C library is
going to do some sort of read() call on the UART/TCPIP
stream. It needs to know to return immediately, not wait
for a fixed-length buffer to be filled.
How can that be done other than insisting on a read of
length 1, and how can that be done other than switching
off buffering with setvbuf?

I posted a C code fragment from an existing NNTP client
that uses 'fgets' and 'fputs' on a line-oriented stream
(associated with a TCP connection). The C library handles
it just fine. There are a half-dozen or more open source
C libraries around that you can download and see how they
do it.
cp->sd = socket(AF_INET, SOCK_STREAM, 0);
if (cp->sd == -1) {
fprintf(stderr, "%s: Unable to create socket: %s\n",
myname, strerror(errno));
return NULL;
}
fprintf(stderr, "Connecting to %s using port %d\n",
cp->hostname, port);
addr.sin_family = hp->h_addrtype;
addr.sin_port = port;
fprintf(stderr, "IP address is %u.%u.%u.%u\n",
(unsigned char)hp->h_addr[0], (unsigned char)hp->h_addr[1],
(unsigned char)hp->h_addr[2], (unsigned char)hp->h_addr[3]);
memcpy((char *)&addr.sin_addr, *hp->h_addr_list, hp->h_length);
diag = connect(cp->sd, (struct sockaddr *)&addr, sizeof(addr));
if (diag == -1) {
fprintf(stderr, "%s: Unable to establish connection: %s\n",
myname, strerror(errno));
close(cp->sd);
return NULL;
}
cp->reader = fdopen(cp->sd, "rb");
if (cp->reader == NULL){
fprintf(stderr, "%s: Unable to reopen file: %s\n",
myname, strerror(errno));
close(cp->sd);
return NULL;
}
cp->writer = fdopen(cp->sd, "wb");
if (cp->writer == NULL){
fprintf(stderr, "%s: Unable to reopen file: %s\n",
myname, strerror(errno));
close(cp->sd);
return NULL;
}
setvbuf(cp->writer, (char *)NULL, _IOLBF, 0);

I think he thinks about reading, not writing.
You read character at time and stop when line ending
is hit... all in all input is buffered anyway, fact
that you do getc is just adjusting pointer in buffer :P

[[e-mail]]

Post by Branimir Maksimovic

Post by Scott Lurndal
cp->reader = fdopen(cp->sd, "rb");
if (cp->reader == NULL){
fprintf(stderr, "%s: Unable to reopen file: %s\n",
myname, strerror(errno));
close(cp->sd);
return NULL;
}
cp->writer = fdopen(cp->sd, "wb");
if (cp->writer == NULL){
fprintf(stderr, "%s: Unable to reopen file: %s\n",
myname, strerror(errno));
close(cp->sd);
return NULL;
}
setvbuf(cp->writer, (char *)NULL, _IOLBF, 0);

I think he thinks about reading, not writing.

Scott posted code for reading earlier:

bp = fgets(buffer, sizeof(buffer), conp->reader);
if (buffer[0] != '2') {
fprintf(stderr, "auth pass returns %s\n", buffer);
}

Post by Branimir Maksimovic
You read character at time and stop when line ending
is hit... all in all input is buffered anyway, fact
that you do getc is just adjusting pointer in buffer :P

My question is about what system call the C library should
be doing. Not just for Unix/Windows/MVS, but an operating
system in general.

I think a C library fopen() should really result in an fopen()
syscall. Why do we have an open() syscall instead of an
fopen() syscall?

Now that I have control over all the components (editor,
news reader, C library, OS), I'd like to rationalize C90.
E.g. maybe I need a #include <c90ext.h> that defines
things like:
ESC_CHAR '\x1b'
ESC_STR "\x1b"
so that I don't need that value hardcoded in micro-emacs.

I'm inclined to say that setvbuf NBF of stdin switches off
character echo. That will allow micro-emacs to work
without extensions.

And setvbuf of non-(stdin/stdout/stderr) needs to be done
for NBF or LBF otherwise you get FBF.

BFN. Paul.

[Branimir Maksimovic]

Post by Scott Lurndal

Post by Branimir Maksimovic

Post by Scott Lurndal
cp->reader = fdopen(cp->sd, "rb");
if (cp->reader == NULL){
fprintf(stderr, "%s: Unable to reopen file: %s\n",
myname, strerror(errno));
close(cp->sd);
return NULL;
}
cp->writer = fdopen(cp->sd, "wb");
if (cp->writer == NULL){
fprintf(stderr, "%s: Unable to reopen file: %s\n",
myname, strerror(errno));
close(cp->sd);
return NULL;
}
setvbuf(cp->writer, (char *)NULL, _IOLBF, 0);

I think he thinks about reading, not writing.

bp = fgets(buffer, sizeof(buffer), conp->reader);
if (buffer[0] != '2') {
fprintf(stderr, "auth pass returns %s\n", buffer);
}

Post by Branimir Maksimovic
You read character at time and stop when line ending
is hit... all in all input is buffered anyway, fact
that you do getc is just adjusting pointer in buffer :P

My question is about what system call the C library should
be doing. Not just for Unix/Windows/MVS, but an operating
system in general.
I think a C library fopen() should really result in an fopen()
syscall. Why do we have an open() syscall instead of an
fopen() syscall?
Now that I have control over all the components (editor,
news reader, C library, OS), I'd like to rationalize C90.
E.g. maybe I need a #include <c90ext.h> that defines
ESC_CHAR '\x1b'
ESC_STR "\x1b"
so that I don't need that value hardcoded in micro-emacs.

I beleive that you have to do that by yourself.
define that things as C does not defines exact represantation
of characters. So you can't match on \r\n rather
on \x0d\x0a ...
Greets, Branimir.

[[e-mail]]

Post by Branimir Maksimovic

Post by ***@gmail.com
Now that I have control over all the components (editor,
news reader, C library, OS), I'd like to rationalize C90.
E.g. maybe I need a #include <c90ext.h> that defines
ESC_CHAR '\x1b'
ESC_STR "\x1b"
so that I don't need that value hardcoded in micro-emacs.

I beleive that you have to do that by yourself.
define that things as C does not defines exact represantation
of characters. So you can't match on \r\n rather
on \x0d\x0a ...

Sorry if I wasn't clear. The character would be different for
different platforms, so an EBCDIC platform would have:

ESC_CHAR '\x27'

Basically for ANSI escape sequences to work at all, we need
an ESC character to exist in the character set, even though the
C90 standard doesn't define one. But we need to define one.

I think it is appropriate to extend C90 to provide some
essentials like this. I think it is quite neat that a C
compiler, editor, news reader can all be written in C90
terms, or C90+ if we add in that extra define. Also for
interactive chat I need a concept of XON, so:

XON_CHAR '\x11'

I might need more characters like DLE to write a file
transfer protocol. I haven't crossed that bridge yet.

BFN. Paul.

[[e-mail]]

I was getting ready to implement my zmodem file transfer
using the new fopen("COM1", "r+b") when I realized that
sometimes you want to read from stdin and write to stdout.
So it is better to encapsulate the opening of the COM port.

And in fact, for some reason you may wish to read from
COM1 and write to COM2, or write to PRT (maybe it is
faster). If the user has the ability to wire these things up,
e.g. maybe the TX pin is broken on COM1, then the software
should probably be flexible enough to support two
comms lines.

On a mainframe this is particularly important as you can't
have a read and write active at the same time. I guess that
means it is not full duplex.

Post by Scott Lurndal
cp->sd = socket(AF_INET, SOCK_STREAM, 0);
if (cp->sd == -1) {
fprintf(stderr, "%s: Unable to create socket: %s\n",
myname, strerror(errno));
return NULL;
}
fprintf(stderr, "Connecting to %s using port %d\n",
cp->hostname, port);
addr.sin_family = hp->h_addrtype;
addr.sin_port = port;
fprintf(stderr, "IP address is %u.%u.%u.%u\n",
(unsigned char)hp->h_addr[0], (unsigned char)hp->h_addr[1],
(unsigned char)hp->h_addr[2], (unsigned char)hp->h_addr[3]);
memcpy((char *)&addr.sin_addr, *hp->h_addr_list, hp->h_length);
diag = connect(cp->sd, (struct sockaddr *)&addr, sizeof(addr));
if (diag == -1) {
fprintf(stderr, "%s: Unable to establish connection: %s\n",
myname, strerror(errno));
close(cp->sd);
return NULL;
}
cp->reader = fdopen(cp->sd, "rb");
if (cp->reader == NULL){
fprintf(stderr, "%s: Unable to reopen file: %s\n",
myname, strerror(errno));
close(cp->sd);
return NULL;
}
cp->writer = fdopen(cp->sd, "wb");
if (cp->writer == NULL){
fprintf(stderr, "%s: Unable to reopen file: %s\n",
myname, strerror(errno));
close(cp->sd);
return NULL;
}
setvbuf(cp->writer, (char *)NULL, _IOLBF, 0);

Shouldn't there exist a function such as fopen4 so that
you can do:

fopen4("COM1", "rb", &file_reader, "PRT", "w", &file_writer);

or:

fopen3("eternal-september:119", "r+b", &file_reader, &file_writer);

and in both cases, you may get NULL for the second file,
which means that the first file needs to be treated as a
"r+b" meaning you need to do your own fseek when
switching between read and write. The reason for this is
to make it easy to add these new functions even if the
existing OS only has the ability to operate on a single
stream.

Any thoughts?

BFN. Paul.

[Rod Pemberton]

On Tue, 12 Oct 2021 18:49:32 GMT

Post by Scott Lurndal

Post by ***@gmail.com
It will open a device, such as "COM1" on PDOS/386 or
"0x10001" on z/PDOS, as "r+b" and start reading characters
until it hits whatever endless-september uses as end-of-line
marker (they are the server, they can do whatever the fuck
they want - I'm pretty sure it is CRLF), and translates from
ASCII to the local character set.

You might want to read through this first.
https://datatracker.ietf.org/doc/html/rfc3977
"The character set for all NNTP commands is UTF-8 [RFC3629].
Commands in NNTP MUST consist of a keyword, which MAY be followed by
one or more arguments. A CRLF pair MUST terminate all commands.
Multiple commands MUST NOT be on the same line. Unless otherwise
noted elsewhere in this document, arguments SHOULD consist of
printable US- ASCII characters. Keywords and arguments MUST each be
separated by one or more space or TAB characters. Command lines MUST
NOT exceed 512 octets, which includes the terminating CRLF pair. The
arguments MUST NOT exceed 497 octets. A server MAY relax these
limits for commands defined in an extension."

Does he really need to support UTF-8? I rather doubt it, as NTTP works
correctly with Telnet which is ASCII.

Regarding the quote above, it would be rather interesting if there were
any commands that were UTF-8 which were not also ASCII ... i.e.,
non-ASCII UTF-8 commands. So, what is the point of specifying the
larger charset of UTF-8 for the commands, but requiring the smaller
ASCII subset of UTF-8 for the arguments? ... Bizarre. Are there any
non-ASCII UTF-8 commands? I don't happen to see any in the spec.
I.e., all NNTP commands appear to be ASCII. Isn't it useless to support
UTF-8 commands if there are none? ... So far, he only needs to support
ASCII, not UTF-8.

According to the remainder of the document, the NTTP RFC standard
requires use of the ASCII subset of UTF-8 and prohibits use of
non-ASCII UTF-8 for most things (quoted below). There is another
exception to that (not quoted below), and it is the description of the
newsgroup, which may use non-ASCII UTF-8. So, it seems there is only
one place, the newsgroup description, for which UTF-8 may actually be
used. Except, if the warnings against using UTF-8 for other portions
of the NNTP protocol are to be actually believed and taken seriously,
"8-bit encodings SHOULD NOT be used because they are likely to cause
interoperability problems", then UTF-8 can't really be used for
newsgroup descriptions either for the same reason. So far, he only
needs to support ASCII, not UTF-8.


"The names of headers MUST be in US-ASCII."

"Header values SHOULD use US-ASCII ..."

"At present, 8-bit encodings, (including UTF-8) SHOULD NOT be used
because they are likely to cause interoperability problems."

"Although this specification allows UTF-8 for newsgroup names, they
SHOULD be restricted to US-ASCII until a successor to RFC 1036
standardizes another approach. 8-bit encodings SHOULD NOT be used
because they are likely to cause interoperability problems."

[[e-mail]]

Does he really need to support UTF-8? I rather doubt it, as NTTP works
correctly with Telnet which is ASCII.

I'm going all-in with the ASCII now, so that it works on
EBCDIC too. See the translation here:

https://sourceforge.net/p/pdos/gitcode/ci/master/tree/src/pdpnntp.c

BFN. Paul.