Ok because you're using optical go into the PS3 settings options go to sound and ensure optical is the choosen option. Then from settings again go into BD option then audio and ensure bitstream is selected. then
From the main PS3 desktop, go to the "Settings" menu, then down to the "Sound Settings" option and select it, then choose "Audio Output Settings".
It will then ask you to "Select the connector on the TV or AV amplifier (receiver).". At this point select HDMI, or RGB what ever you are using then select Automatic. It will then give you a list of the settings it has selected and ask you to press X to save them. The full list should be the below, but because you will have selected Bitstream just X all the options the PS3 gives you
Dolby Digital 5.1 Ch.
DTS 5.1 Ch.
Linear PCM 2 Ch, 44.1 kHz
Linear PCM 2 Ch, 88.2 kHz
Linear PCM 2 Ch, 176.4 kHz
Linear PCM 2 Ch, 48 kHz
Linear PCM 2 Ch, 96 kHz
Linear PCM 2 Ch, 192 kHz
Linear PCM 5.1 Ch. 44.1 kHz
Linear PCM 5.1 Ch. 88.2 kHz
Linear PCM 5.1 Ch. 176.4 kHz
Linear PCM 5.1 Ch. 48 kHz
Linear PCM 5.1 Ch. 96 kHz
Linear PCM 5.1 Ch. 192 kHz
Linear PCM 7.1 Ch. 44.1 kHz
Linear PCM 7.1 Ch. 88.2 kHz
Linear PCM 7.1 Ch. 176.4 kHz
Linear PCM 7.1 Ch. 48 kHz
Linear PCM 7.1 Ch. 96 kHz
Linear PCM 7.1 Ch. 192 kHz
Hopefully this makes a bit more sense, and fixes the problem.
Remember setting the PS3 to bitstream will mean that your receiver will give you DD not PCM 5.1 as you cannot get PCM 5.1 via optical, you need HMDI