Project

General

Profile

Actions

Bug #10276

closed

Corrupt Radio Reference Importing Due to Not Properly Sanitizing Data

Added by C H over 1 year ago. Updated about 1 year ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
-
Target version:
Start date:
01/15/2023
Due date:
% Done:

100%

Estimated time:
Chirp Version:
next (py3)
Model affected:
all
Platform:
Windows
Debug Log:
I read the instructions above:

Description

Specific symbol I am having issues with is

Duplicate error by
attempting to import zip 27410
copy all data
go to file new
paste all data overwriting channel 0
export csv and save
now attempt to open saved file and you receive an error "unknown file format"

the problem is with field 90
GSO1 V.R.E.T.’S

To make the file load you have to delete the symbol it takes a long time to manually find problems and sanitize files with this error

Also another "bug" is when you import radio reference data the comments column does not show up unless you copy/paste the data to a new file

I also with comments data would be present alongside radio img files but it is not


Files

QueryResult.csv (12.3 KB) QueryResult.csv C H, 01/16/2023 02:40 AM
chirp_debug-uqj5qarh.txt (36.8 KB) chirp_debug-uqj5qarh.txt C H, 01/16/2023 02:40 AM
Actions #1

Updated by C H over 1 year ago

C H wrote:

Specific symbol I am having issues with is

Duplicate error by
attempting to import zip 27410
copy all data
go to file new
paste all data overwriting channel 0
export csv and save
now attempt to open saved file and you receive an error "unknown file format"

the problem is with field 90
GSO1 V.R.E.T.’S

To make the file load you have to delete the symbol it takes a long time to manually find problems and sanitize files with this error

Also another "bug" is when you import radio reference data the comments column does not show up unless you copy/paste the data to a new file

I also with comments data would be present alongside radio img files but it is not

ETA: problem is field 90 comment section

Actions #2

Updated by Dan Smith over 1 year ago

  • Target version set to chirp-py3

I cannot reproduce the copy/paste problem, all the channels paste cleanly for me, including the channel you mention. Can you please attach a debug log? See How_To_Report_Issues.

The missing comment field is a problem, I will fix that. Thanks!

Actions #3

Updated by C H over 1 year ago

Platform is Windows 10

When I export my QueryResults to csv then attempt to open them it says "Unknown File Format".

Everything looks like it worked fine, the problem is when you try to OPEN the file.

The symbol above being present in the comments field is what is creating the issue. When I remove the symbol it works.


[2023-01-15 20:34:54,171] chirp.drivers.puxing_px888k - DEBUG: The file size does not match.
[2023-01-15 20:34:54,171] chirp.drivers.tk8102 - DEBUG: b'54.2'
[2023-01-15 20:34:54,171] chirp.drivers.tk8102 - DEBUG: b'54.2'
[2023-01-15 20:34:54,171] chirp.drivers.tk8102 - DEBUG: b'54.2'
[2023-01-15 20:34:54,171] chirp.drivers.tk8102 - DEBUG: b'54.2'
[2023-01-15 20:34:54,171] chirp.wxui.common - ERROR: <function ChirpMain.open_file at 0x00000237740B1360>: Unknown file format

Actions #4

Updated by C H over 1 year ago

I went back further in debug log
`
[2023-01-15 20:34:23,618] chirp.wxui.query_sources - INFO: Starting QueryThread for
[2023-01-15 20:34:31,924] chirp.sources.radioreference - DEBUG: Mode 5 (Telm) is unsupported
[2023-01-15 20:34:54,171] chirp.chirp_common - DEBUG: Image data has no metadata blob
[2023-01-15 20:34:54,171] chirp.drivers.boblov_x3plus - DEBUG: Boblov_x3plus: match_model: size mismatch
[2023-01-15 20:34:54,171] chirp.drivers.fd268 - DEBUG: Unknowd Feidaxing radio, ID:
[2023-01-15 20:34:54,171] chirp.drivers.fd268 - DEBUG: 000: 4c 6f 63 61 Loca....

[2023-01-15 20:34:54,171] chirp.drivers.fd268 - DEBUG: Unknowd Feidaxing radio, ID:
[2023-01-15 20:34:54,171] chirp.drivers.fd268 - DEBUG: 000: 4c 6f 63 61 Loca....

[2023-01-15 20:34:54,171] chirp.drivers.fd268 - DEBUG: Unknowd Feidaxing radio, ID:
[2023-01-15 20:34:54,171] chirp.drivers.fd268 - DEBUG: 000: 4c 6f 63 61 Loca....

[2023-01-15 20:34:54,171] chirp.drivers.fd268 - DEBUG: Unknowd Feidaxing radio, ID:
[2023-01-15 20:34:54,171] chirp.drivers.fd268 - DEBUG: 000: 4c 6f 63 61 Loca....

[2023-01-15 20:34:54,171] chirp.drivers.fd268 - DEBUG: Unknowd Feidaxing radio, ID:
[2023-01-15 20:34:54,171] chirp.drivers.fd268 - DEBUG: 000: 4c 6f 63 61 Loca....

[2023-01-15 20:34:54,171] chirp.drivers.fd268 - DEBUG: Unknowd Feidaxing radio, ID:
[2023-01-15 20:34:54,171] chirp.drivers.fd268 - DEBUG: 000: 4c 6f 63 61 Loca....

[2023-01-15 20:34:54,171] chirp.drivers.fd268 - DEBUG: Unknowd Feidaxing radio, ID:
[2023-01-15 20:34:54,171] chirp.drivers.fd268 - DEBUG: 000: 4c 6f 63 61 Loca....

[2023-01-15 20:34:54,171] chirp.drivers.fd268 - DEBUG: Unknowd Feidaxing radio, ID:
[2023-01-15 20:34:54,171] chirp.drivers.fd268 - DEBUG: 000: 4c 6f 63 61 Loca....

[2023-01-15 20:34:54,171] chirp.drivers.fd268 - DEBUG: Unknowd Feidaxing radio, ID:
[2023-01-15 20:34:54,171] chirp.drivers.fd268 - DEBUG: 000: 4c 6f 63 61 Loca....

[2023-01-15 20:34:54,171] chirp.drivers.ftlx011 - DEBUG: Drivers's ID string:
[2023-01-15 20:34:54,171] chirp.drivers.ftlx011 - DEBUG: ['R', '\x16\x90']
[2023-01-15 20:34:54,171] chirp.drivers.ftlx011 - DEBUG: Radio's ID string:
[2023-01-15 20:34:54,171] chirp.drivers.ftlx011 - DEBUG: 000: 4c 6f 63 61 Loca....

[2023-01-15 20:34:54,171] chirp.drivers.ftlx011 - DEBUG: Drivers's ID string:
[2023-01-15 20:34:54,171] chirp.drivers.ftlx011 - DEBUG: ['P', '!@']
[2023-01-15 20:34:54,171] chirp.drivers.ftlx011 - DEBUG: Radio's ID string:
[2023-01-15 20:34:54,171] chirp.drivers.ftlx011 - DEBUG: 000: 4c 6f 63 61 Loca....

[2023-01-15 20:34:54,171] chirp.drivers.ftlx011 - DEBUG: Drivers's ID string:
[2023-01-15 20:34:54,171] chirp.drivers.ftlx011 - DEBUG: ['T', 'G\x90']
[2023-01-15 20:34:54,171] chirp.drivers.ftlx011 - DEBUG: Radio's ID string:
[2023-01-15 20:34:54,171] chirp.drivers.ftlx011 - DEBUG: 000: 4c 6f 63 61 Loca....

[2023-01-15 20:34:54,171] chirp.drivers.ftlx011 - DEBUG: Drivers's ID string:
[2023-01-15 20:34:54,171] chirp.drivers.ftlx011 - DEBUG: ['\', 'E\x10']
[2023-01-15 20:34:54,171] chirp.drivers.ftlx011 - DEBUG: Radio's ID string:
[2023-01-15 20:34:54,171] chirp.drivers.ftlx011 - DEBUG: 000: 4c 6f 63 61 Loca....

[2023-01-15 20:34:54,171] chirp.drivers.puxing_px888k - DEBUG: The file size does not match.
[2023-01-15 20:34:54,171] chirp.drivers.tk8102 - DEBUG: b'54.2'
[2023-01-15 20:34:54,171] chirp.drivers.tk8102 - DEBUG: b'54.2'
[2023-01-15 20:34:54,171] chirp.drivers.tk8102 - DEBUG: b'54.2'
[2023-01-15 20:34:54,171] chirp.drivers.tk8102 - DEBUG: b'54.2'
[2023-01-15 20:34:54,171] chirp.wxui.common - ERROR: : Unknown file format
`

Actions #5

Updated by Dan Smith over 1 year ago

  • Status changed from New to In Progress
  • Assignee set to Dan Smith

I see now, I thought you were saying it failed during copy/paste, I understand now, thanks.

Actions #6

Updated by Dan Smith over 1 year ago

As I asked, please attach the full debug.log. Also attach the CSV file that you save out which fails to re-open. I need to see both attached to this bug.

I am still unable to reproduce this, even copy, paste, save as CSV and then re-open.

Actions #8

Updated by Dan Smith over 1 year ago

Okay, I can reproduce this, but only on Windows, not other platforms. This is a unicode encoding problem, not sanitization of special characters. I'll work on this.

Actions #9

Updated by C H over 1 year ago

Yeah I actually noticed the characters were properly sanitized of the symbols when I opened the csv file with a different program right before sending the logs. Looking forward to the comments fix. Is there a way to have comments on your img files also? And another good feature would be setting TX power on CSV files because copying a csv file to a radio img I have to manually set all of my TX power settings.

I primarily use Linux myself but I have been on a different machine than I would normally use, Windows always creates issues with things for some reason...

Actions #10

Updated by Dan Smith over 1 year ago

Yeah I actually noticed the characters were properly sanitized of the symbols when I opened the csv file with a different program right before sending the logs. Looking forward to the comments fix.

The comments fix is queued for tomorrow, so you'll have it then.

Is there a way to have comments on your img files also?

Not currently, as the image file is a direct 1:1 memory map of the radio. So, if the radio doesn't support it, we don't in the image.

And another good feature would be setting TX power on CSV files because copying a csv file to a radio img I have to manually set all of my TX power settings.

It's been on the list for a while.

Actions #11

Updated by C H over 1 year ago

Sounds good, I will be on the lookout and I appreciate the prompt response.

Actions #12

Updated by Dan Smith over 1 year ago

  • Status changed from In Progress to Closed
  • % Done changed from 0 to 100
Actions #13

Updated by C H about 1 year ago

I don't know if you check closed issues but its amazing you fixed the issue this fast and implemented the features I was requesting. Thank you for doing so.

I can confirm that the unicode bug is gone.

But when it comes to the new feature of the power level being present in a csv sheet the results aren't exactly as I imagined although perhaps it is according to your development model.

I imagined setting the legal limits of all of the frequencies in a csv and copying and pasting the list to different radios and the configs basically set themselves for power level.

Just an imaginary example for testing not any real legal limits
lets say uv-5r has
high 4w
low 1w
ch1 has max legal power level of 2w set in the csv

what i imagine is that when you copy ch1 into the radio config, since 4w is higher than 2w that means you would need to automatically drop to the low power to be legal but this is not what happens,

in testing what happens is when i copy a 2w channel it sets the power to high(which on the radio is 4w), in my opinion it should set the power to low(1w) since that radios setting on high(4w) will exceed 2w(limit set in csv)

also I think there should be a .5w setting because of the top of my head at least certain frs channels still has a .5w limit

anyways if this was not your vision for how it should work I understand but I just figured I would give some constructive feedback

Actions #14

Updated by C H about 1 year ago

also i think vhf marine is 25w, there are a lot of random numbers that aren't in the list

Actions #15

Updated by Dan Smith about 1 year ago

Some of what you describe seems like it should be working. Either way, let's please discuss the power level stuff in a new bug unrelated to this unicode issue. Just open one and either I can fix/tweak stuff or explain why it has to be.

Actions

Also available in: Atom PDF