[LispM-Hackers] Fep

[James A. Crippen]

On Fri, 16 Mar 2001, John Morrison wrote:
 
> Didn't want to post anything remotely negative to the list.

Cool.  But I want us to be more open, so I've posted this back.  Don't
worry about infighting and negativity -- we're all fairly reasonable
people with similar goals.  I'm not too worried about the occasional
dispute, just as long as it's resolved cleanly and sensibly.

For instance I think that since there isn't any Lord High Hacker in charge
of this project that we'd best work on being as cooperative as possible.  
That means airing all of our dirty laundry in public.  But it also means
that we have to work at keeping personality conflicts to a minimum.  I
don't want to live with something like the linux-kernel mailing list which
is all personality disputes about who's a better hacker and whose design
philosophy is better.

Oh, and open communication prevents subcabals from forming within our
cabal.  That would cause politics, which I wish to avoid.

> "James A. Crippen" wrote:
> > I've just started stubbing in a Front End Processor.  'fep.cc' and 'fep.h'
> > are where it will live, and I'm looking at gluing it into main() just
> > after the call to e3App::app().  The basic concept I have is that after
> > the e3App stuff finishes in main() then the Fep will start in its
> > loop.  From there the user can init the system (which is just doing that
> > 'new' alloc), load a band, and enter the run loop.
> 
> Coupl'a questions:
> 
> (0) Is this the most important task you could possibly do (in terms of
> feature-completeness) right now?  Is it absolutely on the critical
> path?  If not, then what else is?

I think it will become very useful as we start having to debug the 
internals of the processor and chase down symbols within the band.  I
believe that it will speed up the debugging cycle, as well as allow for
more exploratory interaction with the guts of the system.

> (1) Is gdb (or your favorite debugger) a good-enough, "free" (in terms
> of labor) substitute for a FEP (at least right now)?

GDB doesn't know anything about the macrocode.  You'd have to parse out
addresses by hand (which are dependent on endianness and operating system
and the state of the loader at run time) and chase symbols down.  No fun.
I don't want to be distracted by cruft bubbling up from the underlying
operating system.  I don't care where the libraries are loaded to, and
where my kernel syscalls go.  I just want to examine the emulated
processor and memory without having to become lost in a twisty maze of
32-bit pointers, all alike.

It's of course possible to extend GDB to handle this sort of thing, but
that's a major undertaking in and of itself.  I don't want to do that.  I
never ever ever want to hack on GDB if I can help it.  I've seen RMS do so
and it's nasty shit.

> > I think that some exception handling should be put together so that
> > anywhere that the system crashes it just drops back into the FEP with some
> > diagnostic info and lets the user probe around and twiddle bits.  If an
> > exception can't be handled by Lisp (which means most of the problems that
> > we'll be dealing with) then it can dump to the FEP so we can examine it.
> 
> (2) Might we not want to "suspend" the thread and treat this like
> another "vanilla" fatal error and handle this in Lisp using the Explorer
> bit-hacking stuff instead?  In other words, robustify the shit out of
> the microcode and handle in macrocode (either via the normal debugger or
> some other thing).  Perhaps there's already a way to do this?  Maybe you
> could ask the list.  I mean, we should write everything we possibly can
> in LISP "above" the microcode line and as little as possible below it. 
> I'm comfortable in C++ but I'd much rather do almost anything in LISP. 
> (Remind me to tell you about our "Design Patterns Study Group" panel on
> iterators last Wednesday where I ranted about not having map* functions
> and built-in lists.)

I'm perfectly happy doing that.  I basically want the FEP to be a
mechanism in place to manage the virtual hardware.  To do the things that
you can do with physical hardware that you'd have to do with heavy-handed
hacking for the emulator.  Creating disks, managing devices, and
inspecting the paused system.

But if you want to do some of this in Lisp (or in macrocode) then that's
fine by me.  But I'm pretty sure that there won't be easy ways to do the
device management part from the macrocode.  Or other similar work.  And
making macrocode to inspect the paused processor would be well-nigh
impossible because it would entail all sorts of introspective work that
would disrupt state unless you copied out the existing state and inspected
that.  And doing introspection in the context of handwritten macrocode
makes me shudder with fear.  Dealing with the crusty TICLOS system is
weird enough.

Hmm... Handling many of the microcode exceptions could probably be done
with already implemented traps.  But when we encounter really nasty logic
bugs in *our* code then the system will lack a mechanism for dealing with
this.  So we make a signal handler that tries hard to trap segfaults and
throw to the FEP which can act as a primitive debugger, at least allowing
you to poke around in memory and get the state of the processor.  This
could also be done by making a separate core analysis tool, and we may
need that anyway for other purposes.  But suffice to say that for the
really awful errors the macrocode processor won't be able to handle
it.  Instead of just exiting with some bogus error code we should drop to
something that allows you to investigate what happened and why.

> > I have a couple of questions regarding portability.  The absolute easiest
> > interface for making command line interpreters, and the one that I have
> > some experience with, is readline.  That's a GNU lib.  It won't be
> > available on all platforms naturally.  Should I use it anyway?  Should I
> > cruft my own parser together?  Or should I try to support both?
> 
> (3) And see, here, this I think is a killer.  I am perfectly comfortable
> that source code goes out to people, but I have, er, *issues* with
> assigning the copyright to RMS (and my understanding is that's what
> happens if we touch so much as a single line of LGPL code, which we'll
> likely need to do to get it to run under bare iron).

Agree.  That was implicitly what I was wondering about.  It's certainly
not hard to write a CLI parser, but I was thinking of escaping the
work.  I could always just make a -D define in the makefile that you turn
on to get readline if you have it.

> (4) Basically, code we *don't* write we *don't* have to port to a nasty,
> nasty, environment (bare iron).  I actually wrote I/O code for the JOS
> effort, but it was a royal pain in the ass to associate PC keyboard scan
> codes with ASCII (or Lispm character set, I guess) values.  Not to
> mention writing a scrolling, VGA, text-mode (simple tty-like) character
> i/o (geezus I had to scroll the ascii values in the fb manually). 
> Forget any snazzy cursor-movement shit.

Hmm...  Good point.  But I'm not particularly worried about supporting
bare metal at this point.  I think it would be easier to work towards that
in the future.  And there's nothing wrong with having build switches.  
We'll undoubtedly have to recode a lot of low level crud to support bare
metal and I have no problem with dealing with that when we get there and
not until.

> Anyways, what I'd love to have help on is a big set of problems that
> absolutely are on the critical path:

Ah, but you see I am still totally unfamiliar with the codebase.  Writing
a simple front end would help me become very familiar very quickly as I
glue in interfaces to the various chunks of functionality.  So the FEP is
not just a gratuitous pile of featurefulness, but is also a way for me to
explorationally become familiar with the current extent of the
system.  The byproduct is a useful interface.  It also gives me a chance
to harmlessly practice my C++ skills which are quite rusty.  I have to
spend one minute in Stroustrup for every three minutes of reading your
code, I think.  Most of the stuff I already get but I'm not familiar with
all the metaphors.  So putting a FEP into place would be a good exercise
for me and keep me from mangling the working parts and pieces too much.

> (a) What's the goddamned memory map look like?   Where are the devices
> and how do we emulate them?  Where are the memory-mapped registers? 
> Right now there's a shitty bunch of if-then-else clauses that implement
> the memory map.  It sucks and needs Design (how feasible is it to
> memory-protect parts of address space and then trap to a special routine
> if we're writing to a memory-mapped I/O thang -- and how portable is
> that?).
> 
> (b) Figuring out how we're going to have an abstract screen thing on the
> various build targets.  Should we do: X-Windows (a no-brainer IMO); SVGA
> lib (in case we want to boot linux level 3 and then write to the FB
> directly? -- how portable is SVGA lib -- can we use it on bare iron --
> what's the license?); NT/Windoze; WxWindows?; Qt?; etc.?

LibGGI is very portable and very generic.  I've looked at it previously
and was quite impressed.  It will do whatever text mode you want, can run
X atop it, and can run in an X window or on a Linux SVGA console.  I
believe there is also a Windows port of some flavor.  X, SVGA console, and
Windows should be sufficient.  LibGGI is also very low level support and
shouldn't have a problem doing nothing but putpixels and simple 2d
polygons for us.

There are other options but I haven't examined them as closely.

So...
 
You see my point though.  If you can dissuade me I'll be happy to abandon
the idea, but I'm fairly certain that some sort of front end will soon
become necessary.  We're not going to emulate the Explorer boot proms
because those were all written in microcode, which we don't have docs for.  
And we could write a FEP in macrocode but it's not exactly a feasible
project for the reasons I outlined above.  It's also an excellent exercise
for me to begin to understand the code base while still actively
contributing.  That's my justification.  This isn't certainly something
I'm going to spend a lot of time on, worry you not.  But I'm not familiar
enough with how your system is designed just yet to jump right in and
start hacking.  I like to have an overall global understanding of the
system before I fuck it all to hell. :)

'james

-- 
James A. Crippen <james@unlambda.com> ,-./-.  Anchorage, Alaska,
Lambda Unlimited: Recursion 'R' Us   |  |/  | USA, 61.2069 N, 149.766 W,
Y = \f.(\x.f(xx)) (\x.f(xx))         |  |\  | Earth, Sol System,
Y(F) = F(Y(F))                        \_,-_/  Milky Way.