[Program] FSYS Opener

[TheAmazingSnivy]

Spoiler (Click to View)

(05-19-2012 05:24 PM)Mighty Jetters Wrote:  ...I was not expecting this! If you figure all this out, will we be able to export the Pokémon with their rigging as .DAEs (as well as .OBJ of course)? That would be a major breakthrough for the Colosseum project. We wouldn't have to use printscreen rippers!
Will it also export all the textures as .PNG (unlike BRRES Viewer that only auto-exports as .TGA)?
Sorry for all the questions, I'm just really excited to see a potential new FSYS Viewer.
I genuinely hope you can crack the format and complete this program, so best of luck to you!

I'm glad I surprised someone else, I was crazy hyped when I saw this.

[Mighty Jetters]

Haha, I'm glad I made you feel good, because I really was happy to see that you suddenly starting working on something like this.
I doubt I can provide much technical assistance, but you have my full support.

[tykel]

Reverse-engineering? What a wonderful opportunity to procrastinate!

So, you still have not explained very clearly what an FSYS file is. A bit of googling led me to believe it is a video/audio file used primarily in Gamecube Pokemon games? Secondly, am I right in saying the FSYS file type is not well documented?

Some searching made me come to this (http://forum.xentax.com/viewtopic.php?f=10&t=8890), some guy has started something similar. You should definitely check this out if you haven't already.
In his pseudo-code, you can see the following structure:

Code:

struct fsys_file_header {
  char magic[4]; //FSYS
  uint32_t t1;
  uint32_t t2;
  uint32_t file_size; //DATA_COUNT?
  uint32_t t3;
  uint32_t t4;
  uint32_t info_offset; //start address of info header
  uint32_t data_offset; //DATA_BASE, start address of data
  uint32_t file_size;
};

struct fsys_info_header {
  uint32_t array_offset;
  uint32_t name_table:
  uint32_t base_offset;
};

struct fsys_fdat_entry {
  uint32_t data_hash;  // CRC-32? You probably know more than me.
  uint32_t offset;
  uint32_t size;
  uint32_t t1;
  uint32_t t2;
  uint32_t z_size;
  uint32_t t3;
  uint32_t t4;
  uint32_t t5;
  uint32_t name_offset
};

/* ETC.... */

You may already have got this far, but hopefully this will get you started in the right way.
And those numbers at offset 06 & 07? Well, they do not seem that important if the other person's doc. is to be believed, as they fit into a temp (ie. dummy) variable. Perhaps some metadata, maybe information about the compression, or file version information.

As for LZSS, that is a classic LZ compression scheme, which basically is an improved LZ77 with less redundant data sent.

I see you are using Windows from your screenshots, but let me say this: there is a reason Linux is called a hacker's OS. There are many little utilities available to encrypt/decrypt strings on the command line, among other things. You should be fine with a good hex editor, and I recommend something to prototype quickly (not C++; a scripting language like python, maybe) for experiments.

Sorry if I have oversimplified or repeated stuff you know already.
Good luck.

EDIT: As per the C++ debate; C++ is a very complex language. It is one of the only languages not to have a context-free grammar representation, because of its many ambiguities. It is also one of the slowest to compile. BUT it is definitely one the most powerful you can use: native, object-oriented, template meta-programming, ... There is a reason to the fact it is the #1 language in video games, high-performance programs, and many other areas.
My 2 cents: perhaps C# would have been a wiser choice for a GUI program. Or at least, the GUI side of it, it is much simpler. C++ is a bit more work, but you seem to have something good already, so carry on, good sir.

[TheAmazingSnivy]

What tykel said. (Click to View)

(05-24-2012 11:08 AM)tykel Wrote:  Reverse-engineering? What a wonderful opportunity to procrastinate!

So, you still have not explained very clearly what an FSYS file is. A bit of googling led me to believe it is a video/audio file used primarily in Gamecube Pokemon games? Secondly, am I right in saying the FSYS file type is not well documented?

Some searching made me come to this (http://forum.xentax.com/viewtopic.php?f=10&t=8890), some guy has started something similar. You should definitely check this out if you haven't already.
In his pseudo-code, you can see the following structure:

Code:

struct fsys_file_header {
  char magic[4]; //FSYS
  uint32_t t1;
  uint32_t t2;
  uint32_t file_size; //DATA_COUNT?
  uint32_t t3;
  uint32_t t4;
  uint32_t info_offset; //start address of info header
  uint32_t data_offset; //DATA_BASE, start address of data
  uint32_t file_size;
};

struct fsys_info_header {
  uint32_t array_offset;
  uint32_t name_table:
  uint32_t base_offset;
};

struct fsys_fdat_entry {
  uint32_t data_hash;  // CRC-32? You probably know more than me.
  uint32_t offset;
  uint32_t size;
  uint32_t t1;
  uint32_t t2;
  uint32_t z_size;
  uint32_t t3;
  uint32_t t4;
  uint32_t t5;
  uint32_t name_offset
};

/* ETC.... */

You may already have got this far, but hopefully this will get you started in the right way.
And those numbers at offset 06 & 07? Well, they do not seem that important if the other person's doc. is to be believed, as they fit into a temp (ie. dummy) variable. Perhaps some metadata, maybe information about the compression, or file version information.

As for LZSS, that is a classic LZ compression scheme, which basically is an improved LZ77 with less redundant data sent.

I see you are using Windows from your screenshots, but let me say this: there is a reason Linux is called a hacker's OS. There are many little utilities available to encrypt/decrypt strings on the command line, among other things. You should be fine with a good hex editor, and I recommend something to prototype quickly (not C++; a scripting language like python, maybe) for experiments.

Sorry if I have oversimplified or repeated stuff you know already.
Good luck.

EDIT: As per the C++ debate; C++ is a very complex language. It is one of the only languages not to have a context-free grammar representation, because of its many ambiguities. It is also one of the slowest to compile. BUT it is definitely one the most powerful you can use: native, object-oriented, template meta-programming, ... There is a reason to the fact it is the #1 language in video games, high-performance programs, and many other areas.
My 2 cents: perhaps C# would have been a wiser choice for a GUI program. Or at least, the GUI side of it, it is much simpler. C++ is a bit more work, but you seem to have something good already, so carry on, good sir.

Well, I didn't want it to sound like I was procrastinating, because I'm not.but you are right about the filetype, I haven't heard of it coming from any other games, and it's not documented well. So they might only use it in Pokémon games with battle systems like the DS and GBA games. So I was thinking that maybe they might continue using it later on. Anyway, I found another link (I lost it when my computer got wiped) and there was a user from that same site that used the username "RandomTBrush." Do You think that it's the same RandomTalkingBrush on The Spriter's Resource? I'm to chicken to ask him. Anyway, I'll have to resaerch Python, because I'm not sure where it's used... I'm pretty sure Blender is the one that does...
Edit: And yes, I did have all of my files backed up, but not the links in my browser... And is Linux not a mac? Is it like Ubuntu?

[puggsoy]

(05-29-2012 02:17 AM)TheAmazingSnivy Wrote:  And is Linux not a mac? Is it like Ubuntu?

Yah. Linux basically means Ubuntu, Kubuntu, and so on.

[tykel]

(05-29-2012 02:17 AM)TheAmazingSnivy Wrote:  Well, I didn't want it to sound like I was procrastinating, because I'm not.but you are right about the filetype, I haven't heard of it coming from any other games, and it's not documented well. So they might only use it in Pokémon games with battle systems like the DS and GBA games. So I was thinking that maybe they might continue using it later on. Anyway, I found another link (I lost it when my computer got wiped) and there was a user from that same site that used the username "RandomTBrush." Do You think that it's the same RandomTalkingBrush on The Spriter's Resource? I'm to chicken to ask him. Anyway, I'll have to resaerch Python, because I'm not sure where it's used... I'm pretty sure Blender is the one that does...
Edit: And yes, I did have all of my files backed up, but not the links in my browser... And is Linux not a mac? Is it like Ubuntu?

Haha, I was talking about myself for the procrastinating bit

[TheAmazingSnivy]

(05-29-2012 01:17 PM)tykel Wrote:  

(05-29-2012 02:17 AM)TheAmazingSnivy Wrote:  Well, I didn't want it to sound like I was procrastinating, because I'm not.but you are right about the filetype, I haven't heard of it coming from any other games, and it's not documented well. So they might only use it in Pokémon games with battle systems like the DS and GBA games. So I was thinking that maybe they might continue using it later on. Anyway, I found another link (I lost it when my computer got wiped) and there was a user from that same site that used the username "RandomTBrush." Do You think that it's the same RandomTalkingBrush on The Spriter's Resource? I'm to chicken to ask him. Anyway, I'll have to resaerch Python, because I'm not sure where it's used... I'm pretty sure Blender is the one that does...
Edit: And yes, I did have all of my files backed up, but not the links in my browser... And is Linux not a mac? Is it like Ubuntu?

Haha, I was talking about myself for the procrastinating bit

Oh, sorry...

[thegameexplorer]

.PNG isn't a good way to go, since it doesn't store alpha channels. Maybe later you can find a way to detect the alpha maps and export automaticly as png.

[Previous]

Last time I checked, PNG had wonderful alpha channel support. Belive me, I know what I am talking about.

(Did you know that even the Windows Bitmap format BMP has full alpha channel support? Yep, right, there are 32bit Bitmaps with ARGB data! Windows uses them a lot internally.)

[Hoeloe]

(06-02-2012 09:24 AM)thegameexplorer Wrote:  .PNG isn't a good way to go, since it doesn't store alpha channels.

Wherever you got this information from, it's wrong. Png stores alpha channels wonderfully. Jpg doesn't support alpha, but as far as I know most other types do.

[tykel]

OP, what progress have you made? Can you open files yet?
Or are you still figuring it out?

[TheAmazingSnivy]

(06-04-2012 12:20 AM)tykel Wrote:  OP, what progress have you made? Can you open files yet?
Or are you still figuring it out?

I'm still figuring it out, my dad has given me almost no time for myself, and it's finals this week. I don't expect to get much done, sorry.

[tykel]

I found this page, which includes a FSYS extractor (gcfsysfd): http://hcs64.com/vgm_ripping.html

That might be a first step in the right direction. The problem should then be reduced to parsing .fdat files.
There are programs like in_cube which play audio extracted from FSYS files.

Now, look at this thread. User revelation tells us that Pokemon Colosseum files are derived from a known archive format used in the "sysdolphin" API from HAL labs.

Which leads us to this thread. Aha! Someone trying to do something similar to you, but for SSBM! This post in particular is informative.

Read through that thread, the OP details his progress and there are some screenies from page 2. (51 pages, beware!)

Hope this helps.

[TheAmazingSnivy]

(06-05-2012 02:21 PM)tykel Wrote:  I found this page, which includes a FSYS extractor (gcfsysfd): http://hcs64.com/vgm_ripping.html

That might be a first step in the right direction. The problem should then be reduced to parsing .fdat files.
There are programs like in_cube which play audio extracted from FSYS files.

Now, look at this thread. User revelation tells us that Pokemon Colosseum files are derived from a known archive format used in the "sysdolphin" API from HAL labs.

Which leads us to this thread. Aha! Someone trying to do something similar to you, but for SSBM! This post in particular is informative. Read through that thread, the OP details his progress and there are some screenies from page 2. (51 pages, beware!)

Hope this helps.

You've been a real help! Thank you for the links. I'll look through them right away.
In_cube is a .dll file, but I don't know what it goes to... anyone?

[tykel]

(06-05-2012 02:56 PM)TheAmazingSnivy Wrote:  

(06-05-2012 02:21 PM)tykel Wrote:  blah

You've been a real help! Thank you for the links. I'll look through them right away.
In_cube is a .dll file, but I don't know what it goes to... anyone?

AFAIK in_cube is a plugin for Winamp, a music player, that's why it's a library (DLL).
Also, make sure you read that last thread I linked, it's a gold mine.