PDA

View Full Version : Psychonauts .PKG File Format Specs


bgbennyboy
05-16-2005, 06:45 PM
I've just finished my Psychonauts .PKG Dumper (http://www.lucasforums.com/showthread.php?s=&threadid=148932) so here are the specs for the .PKG file format.


Psychonauts .PKG File Format:
-----------------------------

Offsets are relative to the beginning of the file unless otherwise stated.

In the xbox version the .pkg file is zlib compressed. There is a 16 byte
header then a compressed zlib archive. The format of this header is:

4 bytes: Header 'ZLIB'
4 bytes: Version
4 bytes: Size of file when decompressed
4 bytes: Size of compressed file (minus the 16 byte header)

Structure of file:
------------------
Section Size (in bytes)

Header: 512 bytes
File records: 16 * Number of Files
Unknown data: Offset of Name Directory - End of File Records
Name directory: Offset of File Types - Offset of Name Directory
File Types directory: End of File Types Dir - Offset of File Types Dir
File data: Rest of file


File Header:
------------
4 bytes: Header 'ZPKG'
4 bytes: Version
4 bytes: End of File Types Directory
4 bytes: Number of files
4 bytes: End of File Records
4 bytes: ?
4 bytes: Offset of Name Directory
4 bytes: Offset of File Types Directory


File Records:
-------------
Each file record is 16 bytes long:
4 bytes: Identifier
4 bytes: Offset of name in Name Directory (relative to start of name dir)
4 bytes: Offset of file data
4 bytes: File data size

The identifier identifies the file type:
256: .asd
1280: .atx
2304: .cam
3328: .dds
4352: .dfs
5376: .eve
6400: .h
6912: .hlps
8192: .ini
9216: .jan
10240: .lpf
11264: .lua
12288: .pba
13312: .plb
14336: .psh
15360: .vsh

I'm fairly sure that the identifiers match up to these file extensions.
These file extensions are found in the File Types Directory.


Name Directory:
---------------
This contains the name of every file (without the file extension).
Each filename is null terminated (00)
To get the filename, read the 'Offset of name in Name Directory' value
from the file record and seek to this value + 'Offset of Name Directory'.
Be aware that some files share the same filename.


File Types Directory:
---------------------
This lists the file extensions of the various files in the .PKG.
Each file extension is null terminated.


Dumping files:
--------------
To dump a file:
1) Read the file record
2) Seek to (Offset of name in Name Dir + Start offset of Name Dir)
3) Read the filename and add the correct file extension according to the
identifier
4) Seek to 'Offset of file data'
5) Copy out 'File data size' bytes

john_doe
05-21-2005, 07:03 PM
Very nice!

I've had a look at the other files:

jan are the animation files, plb seem to be the models.
hlps, psh and vsh are Pixel/Vertex Shader programs.

If time permits I'll try to work on the plb files :)

bgbennyboy
05-22-2005, 09:49 AM
Cool! If you or anyone else could figure out the format of the level pack files that'd be great too. The .apf files are the same on both pc and xbox, but the .ppf ones differ (when decompressed).

I'm not sure about the .ppf's at all, I dont *think* they are based on chunks but then that would probably mean that the indexes of the files were stored in the .apf files, but this cant be right because these are the same on pc and xbox, whereas the .ppf files differ.

In common.ppf the first file block refers to textures\icons\ui_icons\ui_joystick_01.dds and if you compare that 'block' to 'ui_joystick_01.dds' from the main .pkg file - its the same data, but without the .dds header data.

In conclusion, I dont really know how the level pack files work. :)

netmonkey
05-22-2005, 02:56 PM
I'll be waiting for PsychonautsRev...

ThunderPeel2001
05-23-2005, 11:46 AM
I'm still waiting for SCUMMRev (well, one that work properly! :))

john_doe
06-03-2005, 09:32 PM
I've had a closer look at the Ppf files. It seems there are several 'sections' of data
- textures (at the beginning of the file, a lot of dds files, "PPAK")
- models (starts with "MPAK" section)
- scripts
- more scripts
- another model.

The texture section is right at the beginning and contains dds textures with a special header which has yet to be analyzed. I couln't find an obvious value in this section which says how big the following texture data is.

The model group starts with the marker "MPAK" and has a rather simple structure, after the marker comes an unsigned short telling the number of model files, then for each file an unsigned short with the length of the filename, then the filename and an unsigned long with the size of the data, after this the model data (same format as the plb files from the main pkg).

Next comes the script section, similar in strucuture to the model section, the only obvious difference that there are compiled Lua scripts here :)

After this section come more scripts, but without the filename.

The last section is some model file, maybe the level itself. It has some data in the "PRCS" section to specify events and Lua functions to be called/used or similar.

bgbennyboy
07-08-2005, 05:49 PM
LF has lost the last month's worth of posts. ::

I've made some progress with the .dds' in the .ppf's, I can now parse many of them, but there are some that still give me headaches. I'll post about what I've found out/what I'm stuck with later :)

bgbennyboy
07-08-2005, 09:42 PM
DDS files inside PPF archives:
'PPAK' - header
2 bytes - Number of DDS files

*Repeat for each DDS*
40 bytes - ?
2 bytes - Filename length
X bytes - Filename
2 bytes - ID number
2 bytes - Bigger ID number
2 bytes - Texture ID
10 bytes - ?
4 bytes - Texture width
4 bytes - Texture height
4 bytes - Number of mipmaps
16 bytes - ?
X bytes - Texture data

I used the information on dds' as set out here. (http://msdn.microsoft.com/library/en-us/directx9_c/directx/graphics/reference/ddsfilereference/ddstextures.asp?frame=true)
TextureID's:
0: Size= (Width*Height)*4
9: Size=(Width * Height) div 2
10: Size=(Width * Height)
11: Size=(Width * Height)

However if the file is a cubemap (it says in the filename) then size:= Size*6

If there are mipmaps then the size of the mipmaps must be taken into account and if the texture is not square then the size must be calculated slightly differently. See the attached file and this (http://msdn.microsoft.com/library/default.asp?url=/library/en-us/directx9_c/directx/graphics/reference/ddsfilereference/ddstextures.asp) for more information.

So far so good, this parses a good few of the files. But problems have occured with 'lightmap000.dds' in asco.ppf:

Texture ID = 0
Width = 256
Height = 128
Mipmaps = 0
Name= textures\lightmaps\asco\lightmap000.dds
Actual size = 174760
Size that tool gives - 131072

The file has no mipmaps, so according to the formula its size should be (256*128)*4. I cant work out a method for getting the size of this texture, the only thing I can think of is that it might perhaps be a volume texture (http://msdn.microsoft.com/library/en-us/directx9_c/directx/graphics/reference/ddsfilereference/ddsvolumetextures.asp?frame=true)

So as things stand, I've made some progress, but am now stuck. The tool always reaches a file with texture id 0 thats either a lightmap or norm (normal map?) and exits.

[Edit] It wouldnt let me attach a file, so i've uploaded it here (http://quick.mixnmojo.com/psychonauts/DDScode.txt)

JustinRoad
07-14-2005, 10:11 AM
Actual size = 174760
Size that tool gives - 131072


I don't actually own psychonauts and haven't looked at the data in question, but I'm a console programmer and this kinda thing really interests me :)

Even though the DDS doesn't report any mipmaps, you are aware that 17460 is very, very close to the total size of the image with mipmaps?

131072 (base level image size) +
32768+
8192+
2048+
512+
128+
32+
8 (the smallest image size as far as I'm aware)

= 174760

EDIT: just to check my math :)

bgbennyboy
07-14-2005, 10:52 AM
Its great to have someone else's input :)

I'm just trying to get my head around all this again, its been a week or so since I was messing with this.

For texture id 0 I've assumed that the minimum mipmap size is 1 because I dont *think* that its one of the DXTx compressions. That could be wrong of course but it works for parsing this file:

textures\leveltextures\as_asylum\as_in_tilefloor_n orm.dds
ID = 442
ID 2 = 45388
Texture ID = 0
Width = 256
Height = 256
Mipmaps = 9
totalsize= 349524

Now I think about it, I actually use the DXTx method for computing the size with textureID 0 - if this really isnt one of the DXTx compressions then the size should be calculated the normal way I think.

This post probably makes no sense, I'm just trying to get my head round my messy code again. 12 bytes is very close, perhaps with lightmaps there are always 8 or 9 mipmaps, even if the mipmap number isnt specified.



[Edit] Ignore all that. It looks like youve got it!

bgbennyboy
07-14-2005, 11:01 AM
Right then it seems I was doing two things wrong:

I was using the DXTx method for calculating the size of non-square textures when (I think) textureID 0 isnt DXTx compression.

When I hardcode 8 mipmaps it parses the first lightmap fine, then computes the size of the next one slightly wrong. I suspect that this is because the next lightmap has 1 less mipmap. Some method of calculating the number of mipmaps from the texture size is probably needed.

Well done JustinRoad! Thank you :)

bgbennyboy
07-14-2005, 11:13 AM
Ok so the next 2 lightmaps:

textures\lightmaps\asco\lightmap001.dds
ID = 483
ID 2 = 45429
Texture ID = 0
Width = 32
Height = 16
Mipmaps = 0
totalsize= 2732
actualsize=2728
=5 mipmaps

textures\lightmaps\asco\lightmap002.dds
ID = 484
ID 2 = 45430
Texture ID = 0
Width = 32
Height = 32
Mipmaps = 0
totalsize= 5456
actualsize=5460
=6 mipmaps

I wouldnt expect lightmap002 to have 6 mipmaps, I'd expect 5 really, so it might be tricky to work out a formula for computing the no of mipmaps.

[Edit] Whoops, 6 mipmaps even
[Edit 2] My mistake - it looks like the no of mipmaps is as on the MSDN page - just with non-square textures its 1 less mipmap.

JustinRoad
07-14-2005, 11:42 AM
12 bytes is very close,


I know. I made a bit of an oopsie in my additions. The 12 bytes is non-existant. The mip size is spot on! (I edited my post to reflect the changes :D)


wouldnt expect lightmap002 to have 4 mipmaps, I'd expect 5 really, so it might be tricky to work out a formula for computing the no of mipmaps.



As far as I can figure out, the general formula for the number of mips is:
NumMips = 1 + LogBase2(Min(width,height))

so, for a 32x32 you will have:
NumMips = 1 + LogBase2(32)
= 1 + 5
= 6 mipmaps

The mips are listed here:
-> 32x32x4 = 4096
-> 16x16x4 = 1024
-> 8x8x4 = 256
-> 4x4x4 = 64
-> 2x2x4 = 16
-> 1x1x4 = 4

The sum of which is 5460...

for a 32x16 you wil have
NumMips = 1 + LogBase2(16)
= 1 + 4
= 5

The mips are listed here:
-> 32x16x4 = 2048
-> 16x8x4 = 512
-> 8x4x4 = 128
-> 4x2x4 = 32
-> 2x1x4 = 8

The sum of which is: 2728

bgbennyboy
07-14-2005, 11:58 AM
Thanks :) but I'm not sure exactly what LogBase2() means. I have very little maths knowledge.

Instead I have done this, which seems to work

if mipmaps=0 then
begin
temp:=max(width, height);
case temp of
512: mipmaps:=10;
256: mipmaps:=9;
128: mipmaps:=8;
64: mipmaps:=7;
32: mipmaps:=6;
16: mipmaps:=5;
8: mipmaps:=4;
4: mipmaps:=3;
2: mipmaps:=2;
1: mipmaps:=1;
end;
end;

if width <> height then dec(mipmaps, 1);


This seems to work with the files that I've tried so far :)
Now it errors with 'flames' textures that all have a width of 1097859072. Nothings every easy...

JustinRoad
07-14-2005, 12:08 PM
Okay, LogBase2 is pretty much what your 'if' statement does, but it does it in a pure 'mathematical' way.

If you know anything about logarithms, you can use the natural log: eg. LogBase2(x) = Log(x) / Log(2);

Your version will work okay for cases where 1 axis is no more than 2 times the size of another axis.

What if your texture was 512x128? According to you it will be 10 mipmaps when in reality it will be 8 mipmaps.

What you really need to do is something like this:

if mipmaps=0 then
begin
case min(width,height) of
512: mipmaps:=10;
256: mipmaps:=9;
128: mipmaps:=8;
64: mipmaps:=7;
32: mipmaps:=6;
16: mipmaps:=5;
8: mipmaps:=4;
4: mipmaps:=3;
2: mipmaps:=2;
1: mipmaps:=1;
end;
end;

where min(x,y) simply returns the smaller of the 2 numbers.
eg. (I haven't done pascal in a while, so excuse poor code)

function min(x : Integer; y: Integer) : Integer;
begin
if (x < y) then
result := x;
else
result := y;
end;

bgbennyboy
07-14-2005, 12:14 PM
Thanks for explaining :)

As you can see, I edited the old post while you were typing that one, but I used max instead of min().

I've just tried min() and it computes the size incorrectly, are you sure you meant min and not max?

[Edit] No, you're right, it is min. I didnt remove the 'if width <> height then dec(mipmaps, 1);' part when I tried it :)

JustinRoad
07-14-2005, 12:27 PM
Well, I'm fairly certain it's min(x,y) :)

For example: The only possible mipmaps on 512x128 are:

512x128
256x64
128x32
64x16
32x8
16x4
8x2
4x1

There are 8 of them. If you were to use max(512,128) your function will return 10, which is incorrect in this case :)

The number that this returns includes the base mipmap level.

So to re-iterate. to calculate the number of mipmaps, this is what my functions would look like (just in case I've confused the issue :D)

function LogBase2(Value: Integer) : Integer
begin
case Value of
512: LogBase2 := 9;
256: LogBase2 := 8;
128: LogBase2 := 7;
64: LogBase2 := 6;
32: LogBase2 := 5;
16: LogBase2 := 4;
8: LogBase2 := 3;
4: LogBase2 := 2;
2: LogBase2 := 1;
1: LogBase2 := 0;
default: { what is the default handler for a case statement? I forgot! :)}
ReportError("This function should only be called with power of 2 values!");
end;

end;

function GetNumMips(Width, Height : integer) : Integer
begin
Smallest : Integer;
NumMips: Integer;

Smallest := min(Width, Height);

GetNumMips := 1 + LogBase2(Smallest);

end;

Did that help?

bgbennyboy
07-14-2005, 12:38 PM
I should make new posts rather than keep editing my old ones. :) This is what I have at the moment, and it seems to work ok.

if (mipmaps=0) or (mipmaps > 20) then
begin
temp:=min(width, height);
case temp of
512: mipmaps:=10;
256: mipmaps:=9;
128: mipmaps:=8;
64: mipmaps:=7;
32: mipmaps:=6;
16: mipmaps:=5;
8: mipmaps:=4;
4: mipmaps:=3;
2: mipmaps:=2;
1: mipmaps:=1
else
Mipmaps:=1;
end;
end;

total:=GetDDSSize(1, Mipmaps, (Width * Height) * 4);


It looks like it does the same as yours, thank you for all your help.
Now it seems that some files have their width/height specified within the dds data, not within the header. Which either means that they have a different (and larger) header to the other files, or..they are just weird :~

bgbennyboy
07-14-2005, 01:52 PM
All texture format 0:

common.ppf
(1st file)
joystick01.dds
64*64
actual size=14628

In almost every other .ppf:

textures\fxtextures\flames\flickera_00.dds
64*64
actual size=60836

flames_00.dds
64*128
actual size=88016

Their second ID No is always 0 and mipmaps always =1. What makes these files special is that their width/height comes straight after the 44 byte header. This means that the 'actual size' values I've used here may be out by 8+ byes as the header may be bigger.

I cant see a way of getting the size from these values. As its texture id 0 it should be (w*h)*4 but this is obviously wrong. For now I just hardcode the sizes for these particular files and seek past them, hopefully there wont be any other files of this type.

[Edit] Unfortunately it seems like quite a few files use this 'other' format.

JustinRoad
07-15-2005, 02:54 AM
thank you for all your help.


You're welcome. It's kewl what you're doing. Keep it up :)


textures\fxtextures\flames\flickera_00.dds
64*64
actual size=60836


I have a suspicion that this format is an 'animated' format - meaning there are several frames of animation in the actual size there. Can you confirm this? ie. is there any reference to flickera_01.dds anywhere in the stream?

In your header might be some sort of animation frame count. I suspect you're looking for either a '0x0d' or '0x03' (depending on the format of the picture)

edit: Also, just to be sure. Is there maybe a number in the stream/header that looks like it's very very close to the 'actual size'. This may help particularly as DDS files have a field dwLinearSize which, if not set to zero, could easily help you skip these files.

It may also be worth looking at the DDS file format. If you have access to the D3DSDK, lookup DDSURFACEDESC2. A DDS file is simply the bytes 'DDS ' followed by a DDSURFACEDESC2 structure followed by the image data. Check out the dwFlags to check whether the CAPS fields, PIXELFORMAT field and (most importantly) the dwFourCC field are set. There is a possibility that this type of file also contains different surfaces (Eg. zbuffer stuff). You may have to start parsing DDS files. If the dwFourCC is not still something like 'DXT1','DXT3','DXT5' you need to find out what it is.

Failing that, you could always post the chunk of data in question somewhere and I could take a look at it :)

bgbennyboy
07-15-2005, 08:41 AM
It looks like they are animations like you said. But the frames are all within the same file. I've only looked at
ui_joystick_01.dds right now but:

The file sizes in my previous post were wrong, in this other format the header is 72 bytes not 44 as before.

Header:
2 bytes - No of frames ?
2 Bytes - ID2 - (always 0)
40 bytes ?
4 bytes - width
4 bytes - height
4 bytes - mipmaps
16 bytes - ?

Then comes a normal block of texture data - the size of this is worked out from the texture ID as usual. I havent yet worked out where in the header this is stored. For ui_joystick_01.dds its texture id 9.

Then for the remaining frames:
A normal 44 byte header
Texture data

bgbennyboy
07-16-2005, 05:55 AM
I can now parse all the .dds's (except a few in common.ppf which I will fix later). The next step is to clean up my messy code and then work out which texture id corresponds to which dds format so I can recreate the dds' :)

bgbennyboy
07-16-2005, 03:32 PM
I spoke too soon. I can parse all the dds' in the pc version, but the xbox version has another texture id - 14.

In asco.ppf (xbox):

textures\lightmaps\asco\lightmap000.dds (yes that file again).
ID = 313
ID 2 = 33119
Texture ID = 14
Width = 256
Height = 128
Mipmaps = 9
Actual size = 44716


I'm having problems working out the size with this texture id.
If its not a DTXx compression:
256 by 128 = 32768
128-by-64 =8192
64-by-32 = 2048
32-by-16 = 512
16-by-8 = 128
8-by-4 = 32
4-by-2 = 8
2-by-1 = 2
1-by-1 = 1
total=43691


If its a DXT compression then it should use the non-square dxt method ( max(1,width 4)x max(1,height 4)x 8 (DXT1) or 16 (DXT2-5) ):
256*128 = 32768
128*64 = 512 * 16 = 8192
64*32 = 128 * 16 = 2048
32*16 = 32 * 16 = 512
16*8 = 8 * 16 = 128
8*4 = 2 * 16 = 32
4*2 = 1 * 16 = 16
2*1 = 1 * 16 = 16
1*1 = 1 * 16 = 16
total=43728

Its not massively off with either method, but obviously its not correct. Maybe JustinRoad has an idea? :)

I've attached the file to this post.

JustinRoad
07-17-2005, 04:53 AM
As far as I can figure out, it is a paletted image.

The breakdown is as follows:

at 0x007e in the file is a 16 bit number. Not sure if this specifies whether there is a palette or not. In the data you gave me this is 0x0001.

at 0x0080 is where the palette begins. Each entry is 4 bytes (r,g,b,a) and there are 256 entries. After this is where the index data begins (I think :D)

So, your actual size is comprised as follows:
44716 =

2 bytes for 16 bit number (is there a palette?)
1024 bytes for palette
32768 bytes for 1st mip
8192 bytes for 2nd mip
2048 bytes for 3rd mip
512 bytes for 4th mip
128 bytes for 5th mip
32 bytes for 6th mip
8 bytes for 7th mip
2 bytes for 8th mip
(there is no 9th mip)

At least I don't think there is a 9th mip. A 256 by 128 image will only have 8 mips so that 9 must mean something else.

How did I figure this out? I looked at the data in my favourite hex editor (Visual C :D) and found the palette chunk quite remarkable (with all those 0xFF's in the alpha channel for all the palette entries).

Let me know if it works. I haven't actually done the extraction myself and checked it out - that's what you're for :)

bgbennyboy
07-17-2005, 02:01 PM
Nice one!

I've not tested it with all the xbox files yet but I can parse through Xbox Asco.ppf now.

The 2 byte value at comes directly after the usual 44 byte header.
If it = 256 or 1 - then there's the 1024 byte palette, otherwise there isnt.

The number of mipmaps is still correct for the most part, however (as with texture id 0) you need to run a check if its a non-square image:

if (mipmaps=0) or (width<>height) then
begin
temp:=min(width, height);
case temp of
512: mipmaps:=10;
256: mipmaps:=9;
128: mipmaps:=8;
64: mipmaps:=7;
32: mipmaps:=6;
16: mipmaps:=5;
8: mipmaps:=4;
4: mipmaps:=3;
2: mipmaps:=2;
1: mipmaps:=1
else
Mipmaps:=1;
end;
end;


Thank you for this, I wouldnt have thought about a palette. :) I'll start testing with the other files now.

I've also made progress on checking which texture id corresponds to which DDS type.
Texture ID:
0 = 32 bit A8R8G8B8 (8:8:8: argb)
9 = DXT1
10 = DXT3 or DXT5
11 = DXT3 or DXT5
12 = ??
14 =
I'm not sure which of the 2 DXT's 10 and 11 correspond to. When I dumped to both DXT3 and DXT5, the image looked the same in both cases.

bgbennyboy
07-17-2005, 07:37 PM
After finding yet another texture id (6) I can now parse all .dds' in .ppf archives for both the pc and xbox version. Thanks for all your help JustinRoad.

Now I have to write some code to generate a dds header and hopefully that should be it. Hopefully being the operative word :)

JustinRoad
07-18-2005, 06:01 AM
When performing my own file writes I usually just use a static global structure to write the header out.

eg. (C++ code)

char gacDDSSignature[] = {'D', 'D', 'S', ' '};

DDSURFACEDESC2 gsDDSHeader =
{
0, // dwFlags
0, // dwWidth
...
..
etc..
}

EBool WriteDDS(EStream *psStream, EPicture *psPicture)
{
gsDDSHeader.dwWidth = psPicture->iWidth;
gsDDSHeader.dwHeight = psPicture->iHeight;
gsDDSHeader.dwLinearSize = psPicture->iDataSize;
// etc...

psStream->Write(gacDDSSignature, 4);
psStream->Write(&gsDDSHeader, sizeof(DDSURFACEDESC2));
psStream->Write(psPicture->pcData, psPicture->iDataSize);
return ETRUE;
}

And then when I want to write a header I setup only the image specifics (in this case dwFlags, dwHeight, dwWidth, pixel format and caps) and write the static header. It saves a crapload of time :)

john_doe
07-18-2005, 03:48 PM
Nice work you two are doing :)

While searching for more information on how to decompile Lua 4 I came across this exact thread as the first hit :)
The deleted posts are still in Google's cache in case you want to copy them:

http://216.239.59.104/search?q=cache:Lc3D2jUiYBcJ:www.lucasforums.com/showthread.php%3Fs%3D%26threadid%3D148943+lua+4+OP _CALL&hl=en&client=firefox-a

bgbennyboy
07-18-2005, 06:03 PM
Excellent, good old google cache :)
I kept experiencing that too, when searching for information on Xbox ADPCM and Xbox wavebanks I kept coming across my own posts.

I've mostly completed the dds' stuff. It now displays almost all of the dds' in the .ppf archives.

There's still a couple of obscure texture id's that need supporting, but I'm not sure what dds format they correspond to so I'll probably leave them for now. I'll post complete dds specs soon.

I added the final version of your adpcm decoder to Psychonauts Explorer and it works perfectly with the Pc Audio. :) It works for a lot of the Xbox Audio too. Some of the Xbox audio is marked as Xbox ADPCM but it appears to be slightly different to the standard. With the old method of using the Xbox codec the sounds wouldnt play, even though the wav header was correct. Now, with your decoder they do play, but with distortion. I'll upload some examples later and perhaps you could take a look at them sometime? Those are pretty much the only audio files that cant be played now :)

john_doe
07-18-2005, 06:09 PM
Yes, I'll have a look at them later.

I also finally found out why the OP_CALL opcode in my Lua decompiler didn't always work...turns out it worked in functions without parameters. As soon as there were parameters I didn't add the to the stack so the OP_CALL argument pointed to some weird stack location...now it works :)
Still need to get this if/loop-stuff done...

JustinRoad
07-19-2005, 03:46 AM
I downloaded the PC demo of Psychonauts and played through it a couple of times and I'm pretty sure it uses bumpmapping. Which means if it compresses these normalmaps there is a good chance that they use one of the 2 major normalmap compression schemes: 3DC and RXGB.

3DC is just 16 bytes per block, but it encodes only 2 channels of the normalmap (each 8 byte block is a DXT5alpha channel style encoding of the Y, X components of the normal in the normalmap respectively. Z is calculated in the pixel shader from the x,y via z=sqrt(1.0 - x*x + y*y)). I'm not sure about whether it uses 3DC on the XBox (as 3DC is an ATI technology)

RXGB is Doom3 style normalmap compression which is just DXT5 compression but with the red channel moved into the alpha channel and the red channel set to 0 . This gives better encoding fidelity for normalmaps provided you can unswizzle in the pixel shader (ie. texcol.r = texcol.a)

Hopefully this can help with your obscure texture ids.

Oh, by the way. I do have decoders written for all the major DXT formats as well as 3DC, RXGB. If you're interested I could run the 'unknown' texture format data through my decoders and see if they produce anything :)

john_doe
07-19-2005, 05:29 AM
Originally posted by JustinRoad
I'm not sure about whether it uses 3DC on the XBox (as 3DC is an ATI technology)

Afaik nVidia licensed the 3DC technology from ATI so it could be possible.
It's worth a try at least :)

bgbennyboy
07-19-2005, 07:28 AM
I've uploaded a zip here (http://www.gorman.btinternet.co.uk/XboxAudio.zip) with some examples of the distorted Xbox Audio and the original data. All the information about the files is in info.txt. :)

JustinRoad - yes that would be helpful. I'll upload some examples of the texture id's later.

The current wip version of the tool can be found here. (http://www.gorman.btinternet.co.uk/PsychonautsExplorerImageTest.exe) Note that if you dump a raw dds file, this includes the 44 (or 72) byte header. So far texture id's 0 (usually rgb8), 9 (dxt1), 10 (dxt3?) and 11 (dxt5) are supported. 'Animated' dds' (id 0), and id's 6, 12 and 14 are not yet supported.

john_doe
07-19-2005, 10:49 AM
I've had a (real) quick look the two files. It seems there's 8-bit audio data at the end of the file, the first blocks seem to be Adpcm-compressed. Strange....

bgbennyboy
07-19-2005, 12:23 PM
Psychonauts .PPF File Format:
By Benjamin Gorman (bgbennyboy) and Benjamin Haisch (John_Doe)

4 bytes: Header 'PPAK'

Texture section
Contains .dds files without the standard header.
2 bytes - Number of DDS files

Repeat for each file:
40 bytes - ?
2 bytes - Filename length
X bytes - Filename
44 byte header:
2 bytes - ID number
2 bytes - ID2 number
2 bytes - Texture ID
10 bytes - ?
4 bytes - Texture width
4 bytes - Texture height
4 bytes - Number of mipmaps
4 bytes - ?
2 bytes - New Texture ID (used only if 'animated' image, see below)
10 bytes - ?
X bytes - Texture data

Size of texture data depends on the Texture ID:
0: Size=(Width*Height)*4
6: Size=(Width*Height)
9: Size=(Width * Height) div 2
10: Size=(Width * Height)
11: Size=(Width * Height)
12: Size=(Width*Height)*2
14: Size=(Width*Height) (+1024 if has a palette)

If the file is a cubemap (see the filename) then:
Size=Size*6

However:
The 44 byte header is larger for some files:

If ID2=0 and TextureID=0 then the file is an animation.
It contains multiple dds images within one file:

Number of frames=ID1
Texture ID=New Texture ID
After the 44 byte header:
4 bytes - Texture width
4 bytes - Texture height
4 bytes - Number of mipmaps
16 bytes - ??
Essentially this format is just a container. To parse this format you use
exactly the same methods as below.
After the first block of texture data there follows 'Number of frames' -1
dds files that are set out as normal, with a 44 byte header and then the data.
Also note that the frames within this file do not count as seperate dds files
(in the texture section header).


If Texture ID=14 then:
2 bytes - HasPalette
If HasPalette = 1 or 256 then image has a palette.
Palette has 256 entries so is always 1024 bytes in size.
After the palette data comes the texture data as usual.


The file size also depends on whether the image has mipmaps.
Each mipmap is one-fourth the size of the previous. So the size of each mipmap
has to be worked out and added to the total size. The method of working out
the size depends on the texture id.
So if texture ID was 0 with 8 mipmaps and size of 256*256 then:
262144
65536
16384
4096
1024
256
64
16
total size = 349520


If the TextureID = 9, 10 or 11 things are slightly different:
These use DXT compression so when working out the size:
If TextureID = 9 {DXT1} then minimum mipmaps size = 8
If TextureID = 10 or 11 {DXT3 + DXT5} then minimum mipmaps size = 16

Furthermore if TextureID = 9, 10 or 11 and the image is non-square (eg 256*128) then
the following method must be used at each mipmap level:
Size = max(1,width 4)x max(1,height 4) x 8(If ID 9) or x16 (If ID 10 + 11)

Annoyingly there's a further layer of complexity:
If textureID = 0 or 14 then the number of mipmaps reported by the file may be wrong.
So for these 2 ID's the number of mipmaps needs to be corrected:

if (mipmaps=0) or (width <> height) then
begin
temp:=min(width, height);
case temp of
512: mipmaps:=10;
256: mipmaps:=9;
128: mipmaps:=8;
64: mipmaps:=7;
32: mipmaps:=6;
16: mipmaps:=5;
8: mipmaps:=4;
4: mipmaps:=3;
2: mipmaps:=2;
1: mipmaps:=1
else
Mipmaps:=1;
end;
end;

Texture ID's Correspond to:
0: 8:8:8: argb
6: ??
9: DXT1
10: DXT3
11: DXT5
12: 8:8:8: rgb ???
14: ??


Model section.
4 bytes - 'MPAK'
2 bytes - Number of model files
Repeat for each file:
*2 bytes - Filename length
*X bytes - Filename
*2 bytes -??
*4 bytes - Size of model data
*X bytes - Model Data

Named scripts section.
Repeat for each file:
*2 bytes - Filename length
*X bytes - Filename
*4 bytes - Size of script data
*X bytes - Script data

Unnamed scripts section.
*4 bytes - Size of script data
*X bytes - Script data

Level section?
Not present in some files (eg common.ppf)
Size = remaining file data

Its a bit of a mess, I'll try and clean it up sometime :)

bgbennyboy
07-20-2005, 12:11 PM
Attached to this post are examples of some of the other texture id's. The type of texture is in the filename: eg. (6)arial_swz.dds

ID 6 is xbox only and seems (only?) to be used in the font dds files.
ID14 are the palleted files
ID12 are (only?) in the 'animated' dds' - as the individual 'frames'. I think this format is probably a 16 bit rgb format - its the right size for it at least.

All the files in the zip have the standard 44 byte header, although the file for ID 12 has the 72 byte header. :)

JustinRoad
07-22-2005, 03:49 AM
Just to let you know that I won't be able to look at those files for a while. I am in crunch mode at work at the moment.

If only you'd posted them earlier :)

bgbennyboy
07-22-2005, 06:18 AM
Ok, no problem.