Skip to content

csv: Inconsistency re QUOTE_NONNUMERIC #74232

Open
Open
csv: Inconsistency re QUOTE_NONNUMERIC#74232
Labels
stdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

Description

@tlotze
tlotze
mannequin
opened on Apr 11, 2017
BPO 30046
Nosy @serhiy-storchaka, @zhangyangyu, @tlotze, @corona10

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2017-04-11.20:41:55.773>
labels = ['type-bug', 'library']
title = 'csv: Inconsistency re QUOTE_NONNUMERIC'
updated_at = <Date 2017-04-19.07:11:22.021>
user = 'https://github.com/tlotze'

bugs.python.org fields:

activity = <Date 2017-04-19.07:11:22.021>
actor = 'corona10'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2017-04-11.20:41:55.773>
creator = 'tlotze'
dependencies = []
files = []
hgrepos = []
issue_num = 30046
keywords = []
message_count = 5.0
messages = ['291516', '291590', '291862', '291864', '291865']
nosy_count = 4.0
nosy_names = ['serhiy.storchaka', 'xiang.zhang', 'tlotze', 'corona10']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue30046'
versions = []

Linked PRs

Activity

  1. tlotze commented on Apr 11, 2017

    @tlotze
    tlotzemannequin
    on Apr 11, 2017
    MannequinAuthor

    A csv.writer with quoting=csv.QUOTE_NONNUMERIC does not quote boolean values, which makes a csv.reader with the same quoting behaviour fail on that value:

    -------- csv.py ----------

    import csv
import io


f = io.StringIO()

writer = csv.writer(f, quoting=csv.QUOTE_NONNUMERIC)
writer.writerow(['asdf', 1, True])

f.seek(0)
reader = csv.reader(f, quoting=csv.QUOTE_NONNUMERIC)
for row in reader:
    print(row)
    $ python3 csvbug.py 
Traceback (most recent call last):
  File "csvbug.py", line 12, in <module>
    for row in reader:
ValueError: could not convert string to float: 'True'

    I'd consider this inconsistency a bug, but in any case something that needs documenting.

  2. tlotze
    added
    stdlibStandard Library Python modules in the Lib/ directory
    type-bugAn unexpected behavior, bug, or error
    on Apr 11, 2017

  3. zhangyangyu commented on Apr 13, 2017

    @zhangyangyu
    zhangyangyu
    on Apr 13, 2017
    Member

    boolean is not quoted since in Python it's a subclass of int so True and False are numeric. This is also the case with numeric objects defining __int__ or __float__ but doesn't get a corresponding string representation.

    Since QUOTE_NONNUMERIC will converts data to float when reading, I think we may force the converting even when writing so the inconsistency would disappear. Or document this limitation.

  4. serhiy-storchaka commented on Apr 19, 2017

    @serhiy-storchaka
    serhiy-storchaka
    on Apr 19, 2017
    Member

    This issue is not easy, it needs a thoughtful design before starting coding. I agree with Xiang's analysis and proposed solutions. But if just convert numbers to float we can get an overflow for large integers or lost precision in case of Decimal. The consumer of the CSV file may be not Python and it may support larger precision than Python float.

    And it is not clear what would be better solution for enums. Should they be serialized by name or by value?

  5. corona10 commented on Apr 19, 2017

    @corona10
    corona10
    on Apr 19, 2017
    Member

    I would like to solve this issue.
    Is there any other way than casting specifically for the bool object?
    (e.g For general ways?)

  6. corona10 commented on Apr 19, 2017

    @corona10
    corona10
    on Apr 19, 2017
    Member

    Oh, I read the Serhiy Storchaka 's comment just right now.

  7. ezio-melotti
    transferred this issue fromon Apr 10, 2022
  8. serhiy-storchaka
    added this to CSV issues 📊on May 19, 2022

  9. aterrel commented on May 21, 2025

    @aterrel
    aterrel
    on May 21, 2025

    I don't think it is reasonable to change the type system and this isn't a type bug perse, more of a difference in type expectations.

    I've added the note to the docs but this is my first doc fix in CPython so not sure if I did it correctly.

  10. serhiy-storchaka
    added a commit that references this issue on May 31, 2025

    pythongh-74232: Add a note about roundtrip of non-float numerics in CSV

    75409da

  11. serhiy-storchaka commented on May 31, 2025

    @serhiy-storchaka
    serhiy-storchaka
    on May 31, 2025
    Member

    Other example is Fraction. 1/2 cannot be converted to float.

    We need a parameter for the reader to control how to convert non-quoted fields to numeric. Something like parse_float in json, but it will work not only with floats.

    We may add also add parameters for the reader to control how to convert numerics and other types to strings.

  12. serhiy-storchaka
    added a commit that references this issue on Jun 2, 2025

    gh-74232: Add a note about roundtrip of non-float numerics in CSV (GH…

    e814f43
  13. miss-islington
    added 2 commits that reference this issue on Jun 2, 2025

    pythongh-74232: Add a note about roundtrip of non-float numerics in C…

    22aac47

    pythongh-74232: Add a note about roundtrip of non-float numerics in C…

    61c23ee
  14. serhiy-storchaka
    added 2 commits that reference this issue on Jun 2, 2025

    [3.13] gh-74232: Add a note about roundtrip of non-float numerics in …

    7e2ebdb

    [3.14] gh-74232: Add a note about roundtrip of non-float numerics in …

    bf0d412
  15. Pranjal095
    added a commit that references this issue on Jul 12, 2025

    pythongh-74232: Add a note about roundtrip of non-float numerics in C…

    6228673
  16. taegyunkim
    added a commit that references this issue on Aug 4, 2025

    pythongh-74232: Add a note about roundtrip of non-float numerics in C…

    20b4b3e
  17. Agent-Hellboy
    added a commit that references this issue on Aug 19, 2025

    pythongh-74232: Add a note about roundtrip of non-float numerics in C…

    96ad803

  18. serhiy-storchaka commented on Aug 3, 2026

    @serhiy-storchaka
    serhiy-storchaka
    on Aug 3, 2026
    Member

    gh-155097 (PR #155099) makes this round-trip possible without changing the default: bool is still a number for quoting purposes, so True is still written unquoted.

    A formatter writes bools in the form the consumer expects, and a converter reads them back:

    BOOLS = {'true': True, 'false': False}
csv.writer(f, quoting=csv.QUOTE_NONNUMERIC,
           formatter=lambda i, v: 'true' if v is True else 'false' if v is False else str(v))
csv.reader(f, quoting=csv.QUOTE_NONNUMERIC,
           converter=lambda i, s: BOOLS[s] if s in BOOLS else float(s))

    The same applies to Fraction, IntEnum and the other types in the note added by GH-134963.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    stdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions