Load 'em up, let's get running

Knowing where to go next with your OS development is as important a question as how to do it. So, where next, after printing our message to the screen? I figured the next task that I'd need to do, unless my entire OS was going to sit in that first 512 bytes of code, was to load some data from my floppy drive and try and run it. Once again, after making this decision, it led to many more questions than I had answers to.

• Where in memory should I load the next chunk of data? - I needed to think about the memory arrangement.
• How was I going to actually present this combined chunk of code to VirtualBox? - I needed to find (or write) a tool to create a basic floppy image
• How do you load data from a floppy? - A look at the BIOS routines that are available in 16-bit real mode.
• How many chunks of data should I load? 
• Should the amount of data loaded be dynamic or fixed?

That's a start with the questions that went through my head, and so what follows, is how I went about resolving some of these questions and making some progress.

Where to load my data

I made the assumption that the answers to these questions weren't all going to get resolved before making a start on the solution. So, if I'm never going to have all the things decided down before I start, then why not start and then go back and modify the stuff I've already written as needed. An iterative (dare I say 'Agile') process.

So I chose my memory map as the first thing to write down. A table of memory addresses and what will live there. It's not a difficult thing to envisage, writing down where all my kernel code and where all the user code would go, but it's a very difficult thing to start. Like other tasks I've spoken about, starting from a clean sheet of paper makes the job seem so much harder. So I started with what I knew: that my code had been loaded by the BIOS into 07C0h.

From there I started to build out a sensible allocation of blocks that should allow this first iteration to at least work, until I was in a position to refactor and fix. I knew, of all the things, this would end up being replaced by a more dynamic and intelligent process of allocation but without at least a stick in the ground, you can't pull yourself up (by those blessed bootstraps) any further.

I documented the layout in Excel, just so that I could have something to refer to and moved on to the next piece, writing my floppy image.

Laying out a floppy

This wan't actually too complicated, once I figured out that it didn't need to actually be 1.44MB big. I figured out that I could make a simple 512 byte file with just one sector (the boot sector) and as long as you didn't try to access any other sector, the VM would simply boot from it quite happily.

So all I did was build a simple Java app that would take a list of binary output files from NASM and stack them all together into a contiguous file. The only subtlety is to pad each input file out to the 512 byte boundary, so that the next file starts on the sector boundary.

I may go into more detail on this tool (or build a better one) in another post.

And the rest

From here, once I had figured out where to put the data that I load from the disk and how to present it to the VM as a 'floppy' image, the rest was pretty much just a case of grabbing the file from the image and loading it into contiguous memory locations.

This is where you start to realize the complexity of making this process of loading a file of arbitrary length. You need to be able to find out (or calculate) how long the file is, you then need to find a location in memory big enough for it to fit, then you need to do it.

And then ... when if the disk contains files that aren't in sequence in the image but rather scattered around the disk, you need to know which sector it starts in and where the other pieces are.

There's still a lot to do.

There Goes 2011

There it goes, never to return...2011

It's almost a year ago that I last posted and unless I get some very fast skates on and code some very quick Assembly in the next week, this is likely to be the only post of 2011. Sorry about that.

That's not to say that I've not thought about and worked on (figuratively) NickOS, but the truth is, that I've made very little real progress.

One of the hurdles that had/has me stopped is what language for the OS. I had originally wanted to write it in Assembly for the sheer novelty, however, I figure that if I'm trying to do at least a half decent job, I need something more productive than Assembly. The trouble is, I also want to use something that embraces some of the thoughts on concurrency that have become popular once again over the past year or so. MVCC (multi version concurrency control) is something that intrigues me a lot, however, I struggle with its use at systems programming. I really like the idea of more 'managed code' like Java/C# over classic systems languages like C as they could provide a more robust product, but are they really a good fit?

Anyway, there's still plenty to think about so 2012 looks like another full on year. Let's hope the troubles in the world begin moving closer to closure rather than further from it.

Have a great Christmas and a very prosperous and joyful 2012.

Checking in again..

So it seems like I've been away for a long time again, and I'm afraid to say that that's true. The festive season was once again upon us, visitors for Thanksgiving and Christmas and of course the continuing pressence of the littl'n has all but removed any time to make code progress on NickOS. That's not to say I haven't been working hard on it though, I've bought a couple of books on hardware, to try and get my head around some of the challenges that lie ahead, and I've been looking at memory management algorithms as that's the next step in my development.

I've also been looking at writing my own language, on which to build NickOS. I always had the intention that the OS should allow for web-like scripting, providing a kind of DOM  (Desktop Object Model) that could be easily hooked in to via a JS like language. But this is more about a language to write NickOS in at a more basic level. I know it's a big lump to bite off and chew, but I don't like C much and I do like the expressiveness of Lisp and the Lisp like languages. I also like the functional approach (though it's taking its sweet time making much sense to me) as the idea that we should generally look to program without side effects, really appeals.

So, I'm hacking around with the syntax and structure of a NickOS language. Half of me likes the Lisp brackets, whilst the other half likes the C/Java style of curly braces. And then, I've also been looking at Python and Haskell with their 'spaces mean something' approach. Not sure I like that, though I can't justify that feeling yet.

So busy busy busy, but very little to show. I've not forgotten about NickOS and I'll be back, hopefully early in the new year, with something more concrete to talk about.

I'll leave with a final thought, and that was about the name of this little project. NickOS got it's name, funnily enough, from my name, Nick, but I'm now thinking of changing it...but perhaps with a link back to the old name. Was thinking about NOKI but that seems to be some kind of phone tool, what-d'ya reckon?

NASM and includes

I've been building a 'system' library, that essentially provides high level functions to do system level things like draw on the screen, read from disk, you know the kind of thing I mean.

Anyhow, the file was getting pretty large and unwieldy, so I wanted to split the code down into sections that could be maintained independently and included into the main system.asm file. NASM provides a nice way to do this so I thought I'd record it here for my own benefit and anyone else's who may stubble across this.

NASM syntax for specifying an include(from within the asm file) is %include and so my system.asm file contains the following line, to include the screen library code

%include "screen.asm"

I chose not to fill up my root directory with all these included library asms so I put them in to a sub directory named 'system', however, NASM complains that it can't find them. This is easily overcome with NASM's -i switch, which allows you to specify a prefix that should be appended to the include name when searching. Since my build script (build.bat) runs in a a directory alongside my source I needed to also allow for the change of directory in there.

My directory structure looks like this

\NickOS\dev\src\system.asm        <- this has the %include

\NickOS\dev\src\system\screen.asm <- the file I want to include

\NickOS\dev\bin\build.bat         <- my build script

The build script, since it's on a different path uses the following to assemble system.asm

>nasm.exe -f rdf ..\src\system.asm -l .\system.lst -o .\system.rdf -i ..\src\system\

When build.bat runs, NASM uses the path specified with i relative to '\NickOS\dev\bin' to locate the include and build the binary.

Checking in..

I wanted to check in on this blog, to ensure that I carried on with my intent of logging my activity as I progress on the development of NickOS, however, the last few weeks have been somewhat hectic. We were recently blessed by the birth of our first child, and whilst that has opened up my life in many ways, it's also taken a negative toll on other activities (like developments and personal projects). It's amazing how working to someone else's schedule destroys that quality, dedicated time that you were once used to getting.

Anyway, all that said, NickOS is stil alive and I'm beginning to find a schedule and some window's of opportunity to work on it, so I'll be back working on this site too.

To let you know my current issues, I'm currently stuck on some loader code that needs to relocate a loaded binary that's been read in from disk. The relocation process essentially needs to rewrite the code that's been read in from disk, so that any jumps to absolute memory locations are resolved to be correct for the particular location in memory where the binary has been loaded. Since the location is not known ahead of the loading process, this needs to be fully dynamic. I'm just not quite getting it to work...!

Which is all good fun, although somewhat frustrating, but it is driving me to develop better debugging tools, that can ultimately become part of NickOS, so the effort isn't wasted.

Until next time!

It started with Hello

So there I had it, some very simply boot code that brought my emulator alive, if only enough to stutter the words 'NickOS v0.1 is loading' and then stall. It didn't die, but it didn't do anything more either. This all sounds very uneventful, and indeed you're right, it is. But it's exciting too, to appreciate just how alone this code is on the machine. There's nothing else running except for those few bytes of hand crafted assembly, no friendly API's, no drivers, no...well you get the idea....there's nothing. At least, not much.

So, once you get to this famed position, the question likely strikes (at least it did me); where to from here? So I looked ahead and tried to figure out what I wanted my OS to do...

...thought...

I emerged from my time of thinking with some ideas:

• I wanted NickOS to run in protected mode, with access to the higher parts of memory and probably using paging too.
• I wanted to load some kind of kernel separate from the boot loader. 
• The kernel should manage memory, maybe allow multitasking, 
• And I needed to read from a disk and draw to the screen. 
• And, of course, I wanted to ultimately write programs that would run on NickOS

And all of a sudden, the chasm between what I had accomplished here, my lonely isolated piece of code, and what I was shooting for seemed incomprehensibly wide. I was immediately faced with more questions than I had answers and so began a few more days of simple thought.

...thought...

So rather than trying to write the entire kernel in one, I would work on each piece separately, one step, one baby step, at a time. And where better place to start, than protected mode.

Hello World

One of the first things that I guess anyone does, as with any other kind of programming, is to get your new code to print something to the screen. Some message, that gives you confidence that you're going in the right direction, that shows that you're you're finally getting it. So given that most people start with 'Hello World' when programming, I figured why not start this blog with that too.

But I'm not looking to write tutorials on assembly, the boot process of the PC platform or BIOS primitives to writing to the screen. What I want to give you, however, is to provide the pointers to resources (good resources) that already exist that would allow the would be OS programmer to get started. The tools that will help a new adventurer into OS development to 'get it', to give you the resources to carry on, to guide you in the right direction of discovery.

So think of this not as a how to for writing a basic boot loader, but rather a how to for getting started in writing your first boot loader. Two sites out there, if you haven't found them already, that I have found and continue to find invaluable are http://www.osdever.net/ and http://wiki.osdev.org/Main_Page. Both of these sites are backed by a bunch of people who know a lot more than me and who are willing to share it with you. Both these sites, also have great forums | forums, which give you access to those same people, should their written work not be enough. Be sure to RTFM|T (read the fine manual|tutorial) before asking questions, but if you have a worthwhile question, you'll find these forums populated by people very willing to help

In addition to forums mentioned above, I have found newsgroups to be very helpful. As yet, I've not posted to these groups but just subscribing and reading the posts on many of the comp.* groups can be very enlightening. (try comp.lang.asm.x86 for starters.)

Here's the first screen of NickOS (verion 0.1 I called it, should probably not even have been that!), running inside Virtualbox. I originally ran it on Bochs, but I don't have it installed any longer and since I use Virtualbox to host other virtual OS's, I now just use it for my OS development too.

Not really 'Hello World' but serves the same purpose!

And that's it for now. I'll start pushing some of these links into a gadget so they're always there.