add full Chinese characters and codepage support

General support questions (e.g. question about device support problems or general application features/workings).

add full Chinese characters and codepage support

Postby urmhd » Sat Feb 22, 2020 1:08 pm

the Unicode version of Smartdvb still cannot display correct Chinese characters at scanned channel names outputs and EPG data. All are garbage characters there. I've already tried different Chinese fonts and Unicode fonts which support Chinese, none will work. And there's no codepage support Chinese for EPG @channel properties settings page. I've also tried manually modify channel names with correct Chinese characters, it is do support Chinese in this way, but there's hundreds of channels in the channel list, and may be changed by vendor often for new channels added and some removed.

Thank you for developed this nice DVB software, hope it can be added with full Chinese support. I can do test when needed.

BTW, the default codepage for Chinese is 936.
Attachments
Channel_name_and _EPG_garbled.png
garbage characters
Channel_name_and _EPG_garbled.png (138.46 KiB) Viewed 4121 times
urmhd
 
Posts: 0
Joined: Sun Feb 16, 2020 3:01 pm

Re: add full Chinese characters and codepage support

Postby DVB-S PL » Sun Feb 23, 2020 7:12 am

Please record and upload some sample full TS, 1 minute should be enough.
Codepage should be "announced" in descriptors in SDT/EIT tables, so if it is not working automatically right now - may suggest that the way of broadcasting is different than in Europe.
DVB-S PL
 
Posts: 16
Joined: Sun Oct 18, 2009 4:30 pm
Location: Poland

Re: add full Chinese characters and codepage support

Postby urmhd » Sun Feb 23, 2020 3:59 pm

DVB-S PL wrote:Please record and upload some sample full TS, 1 minute should be enough.
Codepage should be "announced" in descriptors in SDT/EIT tables, so if it is not working automatically right now - may suggest that the way of broadcasting is different than in Europe.


It's too large file size to upload the recorded ts file. There's no Codepage but only language is "announced" in descripors of SDT/EIT tables.

Channel name = ServiceName in SDT table, and EPG is in EIT table.
Attachments
EIT_Table.png
EIT table
EIT_Table.png (16.56 KiB) Viewed 4081 times
SDT_Table.png
SDT table
SDT_Table.png (16.9 KiB) Viewed 4083 times
urmhd
 
Posts: 0
Joined: Sun Feb 16, 2020 3:01 pm

Re: add full Chinese characters and codepage support

Postby urmhd » Sun Feb 23, 2020 4:08 pm

to compare, a very old DVB software AltDVB which is not updated for more than 10 years, support Chinese characters well in Channel names and EPG, but unfortunately it doesn't support new hardware CAMs and hevc codec to decode 4K video.
Attachments
Altdvb_EPG.png
Altdvb_EPG.png (53.99 KiB) Viewed 4079 times
urmhd
 
Posts: 0
Joined: Sun Feb 16, 2020 3:01 pm

Re: add full Chinese characters and codepage support

Postby SmartDVB » Sun Feb 23, 2020 7:44 pm

Good day,

the Unicode version of Smartdvb still cannot display correct Chinese characters at scanned channel names outputs and EPG data. All are garbage characters there. I've already tried different Chinese fonts and Unicode fonts which support Chinese, none will work. And there's no codepage support Chinese for EPG @channel properties settings page. I've also tried manually modify channel names with correct Chinese characters, it is do support Chinese in this way, but there's hundreds of channels in the channel list, and may be changed by vendor often for new channels added and some removed.


Did the ANSI version before the unicode update work better with Chinese? Does the windowed EPG viewer (channels > program guide viewer) show the correct Chinese characters? Perhaps this is some unicode conversion bug. Did you try scanning the channels again and removing your old smarteit.db epg database? Unicode build converts EIT data to UTF8 in database so older smarteit.db ANSI databases should be removed to let the data repopulate (but this is probably not the issue). Can you perhaps upload the transport stream sample to some upload site instead of uploading it here locally upload it to some publicly accessible upload site (if that's not too difficult from China). You can also make a transport stream recording perhaps and remove the actual video/audio PID from the recording but make sure you select PAT - PMT - SDT/BAT - EIT options when recording the file (this might create a smaller, uploadable transport stream recording file which i can use to see why or if the parsing code fails somewhere when using chinese DVB eit codepage).

When i install a chinese font (window > fonts folder) i'm able to see a Chinese codepage selection option with the EPG font dialog (settings > EPG). Saving works but there's a bug it seems when reloading the epg items font dialog (but still the correct unicode chinese font should be saved/loaded correctly).

best regards
SmartDVB
Site Admin
 
Posts: 616
Joined: Sun Feb 01, 2009 5:18 am

Re: add full Chinese characters and codepage support

Postby urmhd » Mon Feb 24, 2020 3:59 pm

Hello,

SmartDVB wrote:Did the ANSI version before the unicode update work better with Chinese? Does the windowed EPG viewer (channels > program guide viewer) show the correct Chinese characters? Perhaps this is some unicode conversion bug. Did you try scanning the channels again and removing your old smarteit.db epg database? Unicode build converts EIT data to UTF8 in database so older smarteit.db ANSI databases should be removed to let the data repopulate (but this is probably not the issue). Can you perhaps upload the transport stream sample to some upload site instead of uploading it here locally upload it to some publicly accessible upload site (if that's not too difficult from China). You can also make a transport stream recording perhaps and remove the actual video/audio PID from the recording but make sure you select PAT - PMT - SDT/BAT - EIT options when recording the file (this might create a smaller, uploadable transport stream recording file which i can use to see why or if the parsing code fails somewhere when using chinese DVB eit codepage).

When i install a chinese font (window > fonts folder) i'm able to see a Chinese codepage selection option with the EPG font dialog (settings > EPG). Saving works but there's a bug it seems when reloading the epg items font dialog (but still the correct unicode chinese font should be saved/loaded correctly).

best regards



Yes, ANSI version works fine with Chinese characters, just tested. I had not updated Unicode version from ANSI version, just begin with Unicode version. I've tried to record a transport stream without video/audio PID, please check attached file. (update, I cannot attach it, so I will send it by email, around 645K size).

I can set fonts at font dialog and choose characters at Unicode version, but there's (settings > EPG) no codepage setting, pls see sreenshot below. In ANSI version the font settings do work, but in Unicode version, it does not.
Attachments
EPG font dialog.png
EPG font dialog
EPG font dialog.png (15.35 KiB) Viewed 4027 times
urmhd
 
Posts: 0
Joined: Sun Feb 16, 2020 3:01 pm

Re: add full Chinese characters and codepage support

Postby SmartDVB » Wed Feb 26, 2020 1:06 am

Thanks for the transport recording. Can you test a SmartDVB.exe build with some changes to see if the unicode codepage issues can be solved? I'm quite sure issue is related to the parser, but my knowledge of the Chinese language is non existent.

best regards
SmartDVB
Site Admin
 
Posts: 616
Joined: Sun Feb 01, 2009 5:18 am

Re: add full Chinese characters and codepage support

Postby urmhd » Wed Feb 26, 2020 1:26 pm

SmartDVB wrote:Thanks for the transport recording. Can you test a SmartDVB.exe build with some changes to see if the unicode codepage issues can be solved? I'm quite sure issue is related to the parser, but my knowledge of the Chinese language is non existent.

best regards


Sure, I will do the test. Thank you.
urmhd
 
Posts: 0
Joined: Sun Feb 16, 2020 3:01 pm

Re: add full Chinese characters and codepage support

Postby urmhd » Sun Mar 01, 2020 4:50 pm

I've tried the test build, it works, Chinese characters showed correctly. But there's a small bug, there's a few most near future EPG items missing. Please see below screenshots.
Attachments
EPG_items_missing.png
EPG viewer window to show the same bug.
EPG_items_missing.png (82.62 KiB) Viewed 3925 times
EPG_now_blank.png
EPG now is blank
EPG_now_blank.png (198.94 KiB) Viewed 3926 times
EPG_bug.png
Please see the current program, next program, and a few are missing.
EPG_bug.png (98.51 KiB) Viewed 3922 times
urmhd
 
Posts: 0
Joined: Sun Feb 16, 2020 3:01 pm

Re: add full Chinese characters and codepage support

Postby urmhd » Sun Mar 01, 2020 4:56 pm

I've tried to go back to 0.5.3.26 beta version, it does not have the EPG items missing bug, of course Chinese characters unreadable again.
urmhd
 
Posts: 0
Joined: Sun Feb 16, 2020 3:01 pm

Re: add full Chinese characters and codepage support

Postby urmhd » Tue Mar 03, 2020 5:24 pm

The EPG items missing bug is gone itself. Very strange, indeed.
Attachments
EPG_OK.png
EPG work just fine now.
EPG_OK.png (96.88 KiB) Viewed 3886 times
urmhd
 
Posts: 0
Joined: Sun Feb 16, 2020 3:01 pm

Re: add full Chinese characters and codepage support

Postby urmhd » Tue Mar 03, 2020 5:34 pm

I've tried to delete smarteit.db that day, so EPG will be received again, but no help. And other DVB software did work OK.
urmhd
 
Posts: 0
Joined: Sun Feb 16, 2020 3:01 pm

Re: add full Chinese characters and codepage support

Postby SmartDVB » Thu Mar 05, 2020 11:48 pm

I've tried to delete smarteit.db that day, so EPG will be received again, but no help. And other DVB software did work OK.


perhaps you just didn't tune the frequency with the correct EPG for those entries after deleting the EPG (usually not all EPG data can be found under a single frequency). Otherwise could still be some bug but i didn't really understand your remark that it worked better with version 0.5.3.26 since that part of the parsing was unchanged, so i would not expect that.
SmartDVB
Site Admin
 
Posts: 616
Joined: Sun Feb 01, 2009 5:18 am

Re: add full Chinese characters and codepage support

Postby urmhd » Fri Mar 06, 2020 5:37 pm

SmartDVB wrote:perhaps you just didn't tune the frequency with the correct EPG for those entries after deleting the EPG (usually not all EPG data can be found under a single frequency). Otherwise could still be some bug but i didn't really understand your remark that it worked better with version 0.5.3.26 since that part of the parsing was unchanged, so i would not expect that.


I've tuned different frequencies, and all frequencies have the same problem. And I've tuned to one frequency many times, which will not fix the problem. And it seems that there's some calculation errors, please see the screenshot of OSD EPG, the left time of current program is very larger than expect. This will not happen with version 0.5.3.26 that day.

Anyway, I cannot recall this problem, it works fine now.
Attachments
OSD_EPG.png
the left time of current program is very larger than expect
OSD_EPG.png (88.28 KiB) Viewed 3827 times
urmhd
 
Posts: 0
Joined: Sun Feb 16, 2020 3:01 pm


Return to Support

Who is online

Users browsing this forum: No registered users and 1 guest

cron