cso support

[sundance]

I've been looking for *.cso (compressed iso) support on DreamShell.
This capability is mentioned here and there, in a few places, and more importantly, there are traces of this support in many places throughout the code base.

However, every time I tried to load a .cso file, it resulted in a crash.

I was lucky enough to receive a hint from @ChristinaDreamcustDC, stating that .cso support might have been dropped in recent versions of DreamShell.
There is, indeed, one short line on the topic in the middle of RC4 release note, firmware section :

> Removed support for CSO in IDE loader to save memory =(

So that's interesting for a few reasons :

1) I'm not using IDE, but a SD Card reader on serial port. From the spelling, one could believe that it should not be impacted by the change on IDE, since after all, within `DS/firmware/isoldr`, there are distinct `ide.bin` and `sd.bin` files.

Well, the Serial Port config is clearly impacted. To confirm, I downloaded RC3, and attempted again to load the *.cso file. This time it worked.

So something broke support of *.cso files for all storage types in RC4.

2) "to save memory"

OK, this one is even more interesting. I've been looking at the code in `modules/isofs` pretty closely, and, provided that the *.cso file is using LZO compression, there is nothing that makes it consume more memory than a direct read on *.iso file. LZO decompression only costs a handful of stack variables, nothing significant.

Note that this is fairly different from zlib compression, where there is a state, which costs memory. I want to underline that I'm targeting LZO-compressed *.cso file as my primary application, precisely because it costs less memory and cpu at decompression time.

3) The code is still there.

As mentioned, I've been looking pretty closely at the code in `isofs`, and a little bit at `isoldr` and `firmware/sd`.
It's pretty clear that the ciso code is still here.
More than that, it's still active, it's not gated, and it ends in the produced binary file, meaning it consumes space, and exposes all expected symbols.

Yet, I've not found where exactly the *.cso support is disabled, if it is.

Presuming I would like to re-enable this support, is there a place to look at ?
Can this support be enabled with a simple build flag at compilation time ?

I'm interested in this topic, and may even provide some patches to improve performance of this code path. That is, if there is a way to test the code changes ....

[kof888]

I personally think that the cso format will take extra time and is not very good for compatibility

[sundance]

*.cso support seems broken directly within `applications/iso_loader`.
It doesn't even reach the point where `sd.bin`, the actual SD Card reader firmware, can start its work.
I suspect that iso_loader tries to read the *.cso file once it's selected in GUI, in order to extract some information, such as the name of the game.
It crashes right there, with ugly traces and core dump.

All versions of the iso_loader app crash the same way since RC4 (while RC3 works correctly).
That means this bug has been out since ~4 years.

[sundance]

The ability to print and read traces would help a lot for debugging.
So far, I'm limited to code reviewing, which can catch a few "obvious" things, but is insufficient for serious debugging.

Anyway, having fixed a nb of minor trivial bugs in iso_loader code path, probably without visible consequences,
I've now reached a part where I suspect the *.cso related crash may lie.

Selecting a *.cso file (or any other type of image) on GUI
triggers `fs_iso_mount()`,
which, after successful loading and mounting, invokes `virt_iso_init_percd()`.
That's where it seems to go wrong for *.cso.
`virt_iso_init_percd()` invokes `get_toc_and_lba()`,
of which 2 versions exist in source code, but one is commented, hence only one is valid.
There is no explanation as to why the other version is commented out though left in the code.

Comparing these 2 versions, the code paths for *.cso look very different.

The new code invokes `get_lba_from_mki()` to determine `lba`.
If it doesn't, it unconditionally set it to 150 (a magic number without a name),
and there is this an associated comment :
// TODO: isofile_find_lba(ifs);

which looks suspicious.

This only matters if `get_lba_from_mki()` fails, but without traces, it's difficult to determine if it's what happens.

[SWAT]

In most cases for repacked ISO images used LBA 150, this is a zero LBA for DC GD-ROM.
If ISO is a part of GDI (track03), then LBA fixed to 45000.
CSO doesn't support in IDE loader since RC4, only SD loader support it.
Is ISO image has another LBA, then I try search MKI string in ISO header, that's leave mkisofs at image creation.
But it's not good way, so I put TODO here for it.

[sundance2]

(30.03.2020 12:41)SWAT писал(а):  CSO doesn't support in IDE loader since RC4, only SD loader support it.

In this post, @SWAT seems to confirm that
CSO support is a thing, although limited to SD loader (presumably SDcard on SPI).

According to my tests, including on recent 0.6.11 loader, CSO doesn't work at all, not even on SPI. It used to work in RC3, but seems broken since RC4.

Considering that the loader code seems to receive some attention and updates these days,
could it be possible to check this part?

It would be a pity that CSO support occupies some precious memory space if, ultimately, it doesn't work. On the other hand, if it's supposed to work, it would be great to confirm it by actually testing it. If there are a few tricks to know to get it working with RC4, we would be glad to know them.

[SWAT]

(18.03.2020 10:47)sundance писал(а):  I've been looking for *.cso (compressed iso) support on DreamShell.
This capability is mentioned here and there, in a few places, and more importantly, there are traces of this support in many places throughout the code base.

However, every time I tried to load a .cso file, it resulted in a crash.

I was lucky enough to receive a hint from @ChristinaDreamcustDC, stating that .cso support might have been dropped in recent versions of DreamShell.
There is, indeed, one short line on the topic in the middle of RC4 release note, firmware section :

> Removed support for CSO in IDE loader to save memory =(

So that's interesting for a few reasons :

1) I'm not using IDE, but a SD Card reader on serial port. From the spelling, one could believe that it should not be impacted by the change on IDE, since after all, within `DS/firmware/isoldr`, there are distinct `ide.bin` and `sd.bin` files.

Well, the Serial Port config is clearly impacted. To confirm, I downloaded RC3, and attempted again to load the *.cso file. This time it worked.

So something broke support of *.cso files for all storage types in RC4.

CISO will be supported only in 0.6.0, for other 0.6.x it's disabled for all loaders. I can made build with CISO support if you want.

(18.03.2020 10:47)sundance писал(а):  2) "to save memory"

OK, this one is even more interesting. I've been looking at the code in `modules/isofs` pretty closely, and, provided that the *.cso file is using LZO compression, there is nothing that makes it consume more memory than a direct read on *.iso file. LZO decompression only costs a handful of stack variables, nothing significant.

Note that this is fairly different from zlib compression, where there is a state, which costs memory. I want to underline that I'm targeting LZO-compressed *.cso file as my primary application, precisely because it costs less memory and cpu at decompression time.

You are absolutely right, just did not take into account one more nuance. The LZO code itself takes up memory ~2 Kb and it's essential.
Also the current implementation of reading lzo-compressed images in the loader requires additional *dynamic* memory, unfortunately. The implementation that does not use additional memory is disabled there were some problems with it.

(18.03.2020 10:47)sundance писал(а):  3) The code is still there.

As mentioned, I've been looking pretty closely at the code in `isofs`, and a little bit at `isoldr` and `firmware/sd`.
It's pretty clear that the ciso code is still here.
More than that, it's still active, it's not gated, and it ends in the produced binary file, meaning it consumes space, and exposes all expected symbols.

Yet, I've not found where exactly the *.cso support is disabled, if it is.

Presuming I would like to re-enable this support, is there a place to look at ?
Can this support be enabled with a simple build flag at compilation time ?

I'm interested in this topic, and may even provide some patches to improve performance of this code path. That is, if there is a way to test the code changes ....

Of course you can! Just uncomment ENABLE_CISO=1 in Makefile.cfg of isoldr firmware. The code is still there, but it doesn't compile when disabled.
It's should be works for all loaders.

(24.11.2020 20:58)sundance2 писал(а):  

(30.03.2020 12:41)SWAT писал(а):  CSO doesn't support in IDE loader since RC4, only SD loader support it.

In this post, @SWAT seems to confirm that
CSO support is a thing, although limited to SD loader (presumably SDcard on SPI).

According to my tests, including on recent 0.6.11 loader, CSO doesn't work at all, not even on SPI. It used to work in RC3, but seems broken since RC4.

Considering that the loader code seems to receive some attention and updates these days,
could it be possible to check this part?

It would be a pity that CSO support occupies some precious memory space if, ultimately, it doesn't work. On the other hand, if it's supposed to work, it would be great to confirm it by actually testing it. If there are a few tricks to know to get it working with RC4, we would be glad to know them.

http://www.dc-swat.ru/forum/thread-2646-...l#pid41022

[sundance2]

Thanks for the new sdloader binary @swat ! Much appreciated !

Unfortunately, when trying to test it, I get into an immediate crash.
And I believe it's unrelated to the loader itself.

Rather, while within the ISO loader program, when merely _selecting_ the *.cso file,
it instantly crashes.
I think it doesn't reach the point where it tries to load the image.

Presumably, ISO loader does something when selecting the image, like analyzing the header,
and maybe it doesn't expect a ziso header.

I vaguely remember that it was the pb I used to identify when trying to debug this topic earlier this year.

Do you experience the same issue on your side ?

[sundance2]

Thanks also for the technical elements describing the behavior of that feature.

Trying to get into some details :

> You are absolutely right, just did not take into account one more nuance. The LZO code itself takes up memory Smile ~2 Kb and it's essential.

2 KB for an LZO decoder feels overkill.
This kind of trivial compression format should be decodable using a ~200 bytes routine.
Granted, hand assembly is likely out of scope,
but given the baseline, I would have hoped that a good compiler would make it < 1 KB.

I expect the compiler to use DCE (dead code elimination),
but even in these cases, there might be some code which is integrated into the package and not especially useful.

Well, I guess it's probably a hard topic, out of your scope.

> the current implementation of reading lzo-compressed images in the loader requires additional *dynamic* memory, unfortunately. The implementation that does not use additional memory is disabled there were some problems with it.

I presume the "implementation that does not use additional memory" would be employing in-place decompression ?
This is a feature for many of these low-complexity formats, allowing decompressed data to overwrite the compressed one previously stored in the same buffer.
In general, there is a need for some margin for it to work properly (aka, the end of the compressed block is not positioned at the end of the decompressed block, but rather a few bytes beyond that).
How much is the margin depend on each format. If it's not known or not documented, it can be generously evaluated. Typically one cache line (32-bytes on SH4) should be enough.

> Of course you can! Just uncomment ENABLE_CISO=1 in Makefile.cfg of isoldr firmware.

I believe I did that when investigating the topic earlier this year.
But it did not solve the issue, as apparently the crash happens before reaching the loader.

[SWAT]

(25.11.2020 09:24)sundance2 писал(а):  Thanks for the new sdloader binary @swat ! Much appreciated !

Unfortunately, when trying to test it, I get into an immediate crash.
And I believe it's unrelated to the loader itself.

Rather, while within the ISO loader program, when merely _selecting_ the *.cso file,
it instantly crashes.
I think it doesn't reach the point where it tries to load the image.

Presumably, ISO loader does something when selecting the image, like analyzing the header,
and maybe it doesn't expect a ziso header.

I vaguely remember that it was the pb I used to identify when trying to debug this topic earlier this year.

Do you experience the same issue on your side ?

No, this problem did not exist before. Which DS version are you using?

(25.11.2020 13:05)sundance2 писал(а):  2 KB for an LZO decoder feels overkill.
This kind of trivial compression format should be decodable using a ~200 bytes routine.
Granted, hand assembly is likely out of scope,
but given the baseline, I would have hoped that a good compiler would make it < 1 KB.

Not only algo used this space, also csio format reading (and some other) code too.

(25.11.2020 13:05)sundance2 писал(а):  I presume the "implementation that does not use additional memory" would be employing in-place decompression ?
This is a feature for many of these low-complexity formats, allowing decompressed data to overwrite the compressed one previously stored in the same buffer.
In general, there is a need for some margin for it to work properly (aka, the end of the compressed block is not positioned at the end of the decompressed block, but rather a few bytes beyond that).
How much is the margin depend on each format. If it's not known or not documented, it can be generously evaluated. Typically one cache line (32-bytes on SH4) should be enough.

Yes it is, but some games had a problem with music (judging by my own comment) with this method and maybe not only with music.
This is ~10 years old code:
https://github.com/DC-SWAT/DreamShell/bl...?ts=4#L791

In general, I just did not deal with ciso support anymore, since the focus is more on the IDE mod, only it can allow all the games to run.
Although, in theory, all games can be run on SD too, but a hack is required.

[sundance2]

(27.11.2020 06:54)SWAT писал(а):  No, this problem did not exist before. Which DS version are you using?

It's supposed to be DreamShell 4.0.0 RC4.
When going into the ISO Loader application, there is a `v0.6.0` mentioned in the bottom right.

Also, for completeness,
I went into the `apps/iso_loader/modules` directory,
there is a file named `app_iso_loader.klf` which I presume is the code for ISO Loader,
it's sized at 35112 bytes,
and `md5sum` gives 1abd5c9e5b38d0004d21e8c8cb2552f6 .

[SWAT]

You get it from release archive or github?