Skip to content

msgpack 1.0.0 changes default string encoding / breaks fluentbit communication #157

Closed

Description

@ptramsey

https://pypi.org/project/msgpack/1.0.0/ -- see the section, "Major breaking changes in 1.0":

use_bin_type=True by default. bytes are encoded in bin type in msgpack. If you are still sing Python 2, you must use unicode for all string types. You can use use_bin_type=False to encode into old msgpack format.

if use_bin_type=False isn't passed to FluentSender in msgpack_kwargs, fluentbit will see that the first element of each received message is not of type MSGPACK_OBJECT_STR, and refuse to enqueue the message. If you have debug output turned on in fluent-bit, you should see it output a message like,

[2020/02/20 21:19:55] [debug] [in_fw] parser: invalid tag format, skip.

I have verified that this happens in fluent-bit 1.3.5, and while I haven't tried it using 1.3.8, the relevant code in that release appears unchanged.

Activity

  1. ptramsey commented on Feb 20, 2020

    @ptramsey
    Author

    I'm not sure what the correct path forward is --- perhaps use_bin_type=False should be included by default in msgpack_kwargs. Alternatively, the dependency could be pinned to python-msgpack <= 0.6.2, since 1.0.0 has degraded performance for python 2.

  2. arcivanov commented on Feb 24, 2020

    @arcivanov
    Member

    I'm going to pin it under 1.0.0 for now as a short term solution. Python 2 being dead is not a terrible concern anymore.

    Thanks for the report!

  3. added a commit that references this issue on Feb 24, 2020
    9172d9e
  4. keuko commented on Jan 15, 2021

    @keuko

    I'm going to pin it under 1.0.0 for now as a short term solution. Python 2 being dead is not a terrible concern anymore.

    Thanks for the report!

    Hello,

    Firstly I would like to say thank you for python-fluent-logger !
    Now, please could I ask if there is a plan to support msgpack > 1.0.0 ? As we have plans in Kolla Openstack to integrate fluent-logger into central_logging of openstack. And newest lower-constraints in openstack is msgpack 1.0.2 ..

    https://review.opendev.org/c/openstack/kolla-ansible/+/755775
    https://review.opendev.org/c/openstack/kolla/+/759855

    I am also packaging python-fluent-logger for debian distro, so can I expect new version ?
    This issue was closed almost one year ago and from your comment pin msgpack to < 1.0.0 was only short solution..

    Thank you very much !
    Michal Arbet ( kevko )

  5. keuko commented on Jan 15, 2021

    @keuko

    There is also pull request for unpin msgpack and some patch here ->
    #171

    Could you please verify and merge ?

  6. arcivanov commented on Mar 11, 2021

    @arcivanov
    Member

    In #173 I'm removing Python 2 compatibility and setting msgpack>1.0 dependency. This way everyone on Python 2 still can use 0.9.x.

  7. added 3 commits that reference this issue on Apr 21, 2021
    0a580c4
    3c31c18
    1ac881f
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions