Kotor I / II file format docs?

[magnusll]

I have recently acquired an interest in Kotor modding.
I was thinking about writing a small utility to help me edit some of the stuff you need to modify. However [due probably to my pathetic search-fu], I can't find some nice, plain documentation of the various formats used in the game files. It is also my understanding that there are some formats which are, as of yet, unknown (i.e. walkmeshes and lightmaps, and partially animation?)

Could someone please give me a few pointers to the relevant webpages / forum threads / whatever?

[tk102]

Here's a good starting place for you manusll.
http://nwn.bioware.com/developers

The above website contains documentation of the GFF and ERF file formats as well as how the GFF format is used for various template files (*.ut*) and dialogs.

****


One the great early pioneers of KotOR modding, eiz, documented some of the undocumented file formats, namely the v2.b 2DA file format and the RIM file format:

For the 2da file format (v2.b)

Originally Posted by eiz

website
2DA Format

2DA is the 2-dimensional array format, used to store a large number of tables pertaining to everything from game rules to graphics and sound definitions. It exists in two forms, 2DA V2.0 and 2DA V2.b; only the latter is described here, as the former is documented by BioWare at the For Developers section on the Neverwinter Nights website.

There is not much to say about the 2DA V2.b header, it's just the string "2DA V2.b" followed by a linefeed character.

Directly following the header is a list of column names. Each name is an ASCII string terminated with a tab character. The end of the list is signified by a null character.

Following the column list is a 32-bit unsigned integer containing the number of rows in the table. This is then followed with another tab-terminated list, containing the row names. Unlike the column list, it has no null terminator.

Following the row list is a two dimensional array, in row major order, containing 16-bit offsets into a data area for each cell in the table. Offsets are relative to the start of the data area. This is followed by 2 unused bytes, and then a data section of null terminated strings.

For the RIM format:

Originally Posted by eiz

website
RIM File Format

The RIM file format, used mainly to store module data, is essentially a condensed version of the ERF format, using 1 table instead of 3 and eliminating the description string 'feature'. It consists of a 120 byte header, a resource index table, and raw resource data.

Header
The RIM header is defined as follows:
Offset Type Name Description
$00..$03 char[4] Signature This is the file signature. It should be "RIM ".
$04..$07 char[4] Version Format version. Currently "V1.0".
$08..$0B unsigned int Unknown_1 Appears to be reserved.
$0C..$0F unsigned int EntryCount The number of resources in the file.
$10..$13 unsigned int OffKeyList File offset to the resource list.
$14..$77 char Reserved Reserved area. Should be zeroed.

Key List
The key list is a struct array, with one entry per resource. Its length is defined in the header by the EntryCount field. The structure follows:
Offset Type Name Description
$00..$0F char[16] ResourceName The resource name, not including the extension. Maximum length is 16 characters, unused characters should be null.
$10..$11 unsigned short ResType The resource type, as defined in Resource Types.
$12..$13 unsigned short ResourceID The resource identifier should be a sequential number, starting at zero, and each resource in the file must have a unique ID.
$14..$15 unsigned short Reserved_1 Reserved. Should be zero.
$16..$19 unsigned int Offset The file offset for the resource data.
$1A..$1D unsigned int Length The resource data length.

Resource Types
ResType Extension Description
$0000 res Misc. GFF resources
$0001 bmp Microsoft Windows Bitmap
$0002 mve
$0003 tga Targa Graphics Format
$0004 wav Wave
$0006 plt
$0007 ini Windows INI
$0008 mp3 MP3
$0009 mpg MPEG
$000A txt Text file
$000B wma Windows Media audio?
$000C wmv Windows Media video?
$000D xmv
$07D0 plh
$07D1 tex
$07D2 mdl Model
$07D3 thg
$07D5 fnt Font
$07D7 lua
$07D8 slt
$07D9 nss NWScript source code
$07DA ncs NWScript bytecode
$07DB mod Module
$07DC are Area (GFF)
$07DD set Tileset (unused in KOTOR?)
$07DE ifo Module information
$07DF bic Character sheet (unused)
$07E0 wok
$07E1 2da 2-dimensional array
$07E2 tlk
$07E6 txi Texture information
$07E7 git Dynamic area information
$07E8 bti
$07E9 uti
$07EA btc
$07EB utc Creature blueprint
$07ED dlg Dialogue
$07EE itp
$07EF btt
$07F0 utt
$07F1 dds
$07F2 bts
$07F3 uts
$07F4 ltr
$07F5 gff Generic File Format
$07F6 fac
$07F7 bte
$07F8 ute
$07F9 btd
$07FA utd
$07FB btp
$07FC utp
$07FD dft
$07FE gic
$07FF gui GUI definition (GFF)
$0800 css
$0801 ccs
$0802 btm
$0803 utm
$0804 dwk
$0805 pwk
$0806 btg
$0807 utg
$0808 jrl Journal
$0809 sav Saved game (ERF)
$080A utw
$080B 4pc
$080C ssf
$080D hak Hak pak (unused)
$080E nwm
$080F bik
$0BB8 lyt
$0BB9 vis
$0BBA rim See RIM File Format
$0BBB pth Path information? (GFF)
$0BBC lip
$0BBD bwm
$0BBE txb
$0BBF tpc Texture
$0BC0 mdx
$0BC1 rsv
$0BC2 sig
$0BC3 xbx
$270D erf Encapsulated Resource Format
$270E bif
$270F key

Welcome to the forums!

[magnusll]

Thanks for the quick reply. I was looking around the various sites / utilities / etc. and as far as I can tell, most of the file formats are already editable using several tools, and for the ones which aren't (i.e. walkmeshes and lightmaps) there's a lack of sufficient info to write an importer/exporter/editor anyway.

It looks like there's a lack of one application which can do everything at once, though. The closest thing would be Fred Tetra's KotorTool, but given the lack of updates and his prolonged absence, it currently seems unlikely it will ever reach completion. A pity.

[magnusll]

So, I've fiddled around with the .key and .bif file documentation and the express edition of Visual C# 2005, and I've managed to write a small, stupid application that is able to build a tree similar to the one you see in Kotor Tool, except at the moment it can only process the chitin.key and the .bif files in the data folder and lets you save any resource file included in those.

Before I go to the deep end: is there even any need for this? Right now, it would take an untold number of hours just to reach the functionality of Kotor Tool. I was thinking about closing the gap by brutally stealing incorporating some external applications into the thing (i.e. no sense rewriting a GFF editor when I can simply launch K-GFF from within the program, provided it's installed), but even so it's a long, long, loooooong way to reach current functionality.
On the plus side, I could release this as open source, so when I get fed up someone else could pick up and continue. Which is what everyone is missing right now with Kotor Tool, I suppose.

I guess my question is... is anyone interested?

[tk102]

What new functionality do you want to provide? Just an open-source KotOR Tool clone? Probably not worth the trouble unless you just want to do for an academic exercise. I tend to prefer individual smaller utilities specific for a particular task rather than the Swiss-army knife approach. If there is a particular function that isn't available with the current tools, the time would probably be better spent building a tool specific to addressing that need, IMO. Of course if KotOR 3 ever gets made, we can't count on Fred returning to add support for it in KT...

Kind of fun though to see the game reveal itself through your program though isn't it?

[magnusll]

Originally Posted by tk102

What new functionality do you want to provide? Just an open-source KotOR Tool clone?

I was mostly thinking about things which cannot be done right now but could be in the future. E.g. mdlops is moving things on the animation front, and maybe someone sooner or later will be able to crack walkmeshes and lightmaps. It would be obviously useful to have an editor able to take advantage of this.
Also, the smaller utilities are nice but I prefer an integrated tool. At least in terms of presentation; i.e. a glorified program launcher which doesn't do anything by itself except launch external editors, but it can at least, say, show all resources associated with a certain area in an organized fashion, autoimport models in Gmax etc.

Originally Posted by tk102

Kind of fun though to see the game reveal itself through your program though isn't it?

Yeah, I started playing around with the resource extractor and read through a few scripts... he. Kinda neat...

[Darth333]

Welcome to the forums

Well it looks like KT is no longer supported by Fred and it would be nice if someone could pick up more or less where he left for future developments. Also choice is always a good thing

Personally I like smaller standalone utilities when I want to work on only one aspect of the game without resorting to "heavy artillery" but for bigger projects, a toolset that can handle most files is definitely a plus and can help maintaining my sanity (making a new module with a gff editor requires an incredible amount of patience).

However, we can't deny that while the new tool would certainly be welcomed by those who are still modding the game, until the rumored 3rd game is released, there will be less kotor mods in the making too in the near future as Kotor 2 is already 2 1/2 yrs old.

[magnusll]

Do you mind sending me the corrections to the .RIM file format as explained the first time by eiz? I was working on the following specs:

RIM File Format

The RIM file format, used mainly to store module data, is essentially a condensed version of the ERF format, using 1 table instead of 3 and eliminating the description string 'feature'. It consists of a 120 byte header, a resource index table, and raw resource data.

Header
The RIM header is defined as follows:
Offset Type Name Description
$00..$03 char[4] Signature This is the file signature. It should be "RIM ".
$04..$07 char[4] Version Format version. Currently "V1.0".
$08..$0B unsigned int Unknown_1 Appears to be reserved.
$0C..$0F unsigned int EntryCount The number of resources in the file.
$10..$13 unsigned int OffKeyList File offset to the resource list.
$14..$77 char Reserved Reserved area. Should be zeroed.

Key List
The key list is a struct array, with one entry per resource. Its length is defined in the header by the EntryCount field. The structure follows:
Offset Type Name Description
$00..$0F char[16] ResourceName The resource name, not including the extension. Maximum length is 16 characters, unused characters should be null.
$10..$11 unsigned short ResType The resource type, as defined in Resource Types.
$12..$13 unsigned short ResourceID The resource identifier should be a sequential number, starting at zero, and each resource in the file must have a unique ID.
$14..$15 unsigned short Reserved_1 Reserved. Should be zero.
$16..$19 unsigned int Offset The file offset for the resource data.
$1A..$1D unsigned int Length The resource data length.

I know however that at least the key list as seen above is wrong. As far as I was able to understand, there's no "Reserved_1" and ResType and ResourceID are unsigned int (i.e. 4 bytes each). Could you confirm this (and maybe add further corrections if you know them)?

[tk102]

Originally Posted by magnusll

Do you mind sending me the corrections to the .RIM file format as explained the first time by eiz? I was working on the following specs:



I know however that at least the key list as seen above is wrong. As far as I was able to understand, there's no "Reserved_1" and ResType and ResourceID are unsigned int (i.e. 4 bytes each). Could you confirm this (and maybe add further corrections if you know them)?

Oh I'm sorry magnusll I completely forgot this documentation was inaccurate as-published... The corrections are listed below:

Header
The RIM header is defined as follows:
Offset Type Name Description
$00..$03 char[4] Signature This is the file signature. It should be "RIM ".
$04..$07 char[4] Version Format version. Currently "V1.0".
$08..$0B unsigned int Unknown_1 Appears to be reserved.
$0C..$0F unsigned int EntryCount The number of resources in the file.
$10..$13 unsigned int OffKeyList File offset to the resource list.
$14..$77 char Reserved Reserved area. Should be zeroed.

Key List
The key list is a struct array, with one entry per resource. Its length is defined in the header by the EntryCount field. The structure follows:
Offset Type Name Description
$00..$0F char[16] ResourceName The resource name, not including the extension. Maximum length is 16 characters, unused characters should be null.
$10..$13 unsigned short ResType The resource type, as defined in Resource Types.
$14..$17 unsigned short ResourceID The resource identifier should be a sequential number, starting at zero, and each resource in the file must have a unique ID.
$18..$1B unsigned int Offset The file offset for the resource data.
$1C..$1F unsigned int Length The resource data length.

[magnusll]

Originally Posted by tk102

Oh I'm sorry magnusll I completely forgot this documentation was inaccurate as-published... The corrections are listed below:

Thanks for the confirmation... and no worry about the original post, it took me all of five minutes with the debugger and an hex editor to figure the changes

I can now extract pretty much anything located into the \data, \modules, \rims and \TexturePacks directories with my little tool... it's probably time to add the capability of doing something with all these files though