Partial, "topologies only" v1.9.3-tplg2 release

I do not follow that partial release. From the packager perspective, it requires an additional step to verify the changes from 1.9.3, add extra package to 1.9.3 and eventually the description refers v1.9.3-tplg1 (unreleased in sof-bin). A very confusing step. It would be much better to release 1.9.4 or 1.9.3.1 with all files and a correct changelog.

It would be much better to release 1.9.4 or 1.9.3.1

Can you explain how v1.9.3.1 is different from v1.9.3-tplg1 and why it is better? We have some constraints with version numbers too.

with all files

Do you mean including .ri firmware binaries? We announced topologies-only releases here:
https://mailman.alsa-project.org/pipermail/sound-open-firmware/2022-February/004360.html

The problem with releasing code again is this:

either the code in the corresponding branch is exactly the same, then users get confused why the version number was incremented
or the code in the corresponding branch has some actual differences, then it requires a costly validation cycle again.

and a correct changelog.

Which changelog is incorrect?

It would be much better to release 1.9.4 or 1.9.3.1

Can you explain how v1.9.3.1 is different from v1.9.3-tplg1 and why it is better? We have some constraints with version numbers too.

It's difficult to compare standard release names like v1.9.3 and the ones with the custom suffixes. RPM has some rules, but adding the new number is always the simple way to distinguish releases (and clear to users in my opinion).

with all files

Do you mean including .ri firmware binaries? We announced topologies-only releases here: https://mailman.alsa-project.org/pipermail/sound-open-firmware/2022-February/004360.html

You ended in the middle of something. If you want to split the packaging, split it (firmwares without topologies and versa vice) with the clear tar ball naming like proposed in the above e-mail. And the older releases should follow the previous style (so start from v1.9.4 or v2.1).

The problem with releasing code again is this:

* either the code in the corresponding branch is exactly the same, then users get confused why the version number was incremented

* or the code in the corresponding branch has some actual differences, then it requires a costly validation cycle again.

Yes, split both things then and write or define the linkage clearly. Note that adding extra version allows you to endlessly update topologies or firmware files when you need make update to your standard release.

and a correct changelog.

Which changelog is incorrect?

https://github.com/thesofproject/sof-bin/releases/tag/v1.9.3-tplg2
https://github.com/thesofproject/sof/releases/tag/v1.9.3-tplg2

I don't see directly a link to v1.9.3-tplg1 (which was unreleased in the sof-bin repo) so this information is problematic.

@lgirdwood

It's difficult to compare standard release names like v1.9.3 and the ones with the custom suffixes. RPM has some rules, but adding the new number is always the simple way to distinguish releases (and clear to users in my opinion).

Understood, we'll see if v1.9.3.1 is possible. Another scheme we considered is v1.9.3a, would that be a problem too?

You ended in the middle of something. If you want to split the packaging, split it (firmwares without topologies and versa vice) with the clear tar ball naming like proposed in the above e-mail.

We haven't released firmware yet since we started discussing this split. Actually we don't expect to make a "firmware-only" release in the future.

And the older releases should follow the previous style (so start from v1.9.4 or v2.1).

Unfortunately we have a customer requesting a topology update in the v1.9 branch and we cannot go back in time and undo earlier and combined v1.9.x releases.

Trying to rephrase and make sure I understand: in other words you don't want to have to combine two tarballs where files from one tarball overwrite files from another? Even if not your preferred description, is that an accurate description of what you don't want?

I don't see directly a link to v1.9.3-tplg1 (which was unreleased in the sof-bin repo) so this information is problematic.

v1.9.3-tplg1 was aborted. I suspect these release notes were automatically generated as a log from -tplg1 to -tplg2. So the fix is hopefully as simple as adding the log from v1.9.3 to v1.9.3-tplg1? If correct @lgirdwood or @keqiaozhang could you look at this?

@lgirdwood

It's difficult to compare standard release names like v1.9.3 and the ones with the custom suffixes. RPM has some rules, but adding the new number is always the simple way to distinguish releases (and clear to users in my opinion).

Understood, we'll see if v1.9.3.1 is possible. Another scheme we considered is v1.9.3a, would that be a problem too?

I did this for ALSA for many years and I was told that one more sub-version number is better for some distros - packaging schemes (unfortunately, I don't remember the affected distro). The comparison of two versions should be consistent and determinable.

But if you don't want to release the full package, v1.9.3.1 may confuse users a lot.

You ended in the middle of something. If you want to split the packaging, split it (firmwares without topologies and versa vice) with the clear tar ball naming like proposed in the above e-mail.

We haven't released firmware yet since we started discussing this split. Actually we don't expect to make a "firmware-only" release in the future.

And the older releases should follow the previous style (so start from v1.9.4 or v2.1).

Unfortunately we have a customer requesting a topology update in the v1.9 branch and we cannot go back in time and undo earlier and combined v1.9.x releases.

Trying to rephrase and make sure I understand: in other words you don't want to have to combine two tarballs where files from one tarball overwrite files from another? Even if not your preferred description, is that an accurate description of what you don't want?

The overwrite is ugly in my eyes - very unusual for the releases - but the main objection is that we don't see changes on the first hit. If 90% of files are similar - why to carry duplicated contents in the package build system?

Could you just release new files or fixes then in a tar ball? The changes and updates will be visible from the first look. From the initial e-mail, my assumption was to have two packages with separate contents.

From the package perspective, it's more straight to add new files or update few files in the package and it helps to determine the testing requirements.

Also, it may be more handy to retain the updated paths - I speak about sof-tplg-v1.9.3 subtree in the v1.9.3 main package. The tar ball tree structure should be unchanged. Also, there are no instructions (and may be script) to copy the files to the right location.

I don't see directly a link to v1.9.3-tplg1 (which was unreleased in the sof-bin repo) so this information is problematic.

v1.9.3-tplg1 was aborted. I suspect these release notes were automatically generated as a log from -tplg1 to -tplg2. So the fix is hopefully as simple as adding the log from v1.9.3 to v1.9.3-tplg1? If correct @lgirdwood or @keqiaozhang could you look at this?

Yes, it's confusing to refer tplg1 from tplg2 when there is no tplg1 release.

@plbossart

The overwrite is ugly in my eyes - very unusual for the releases - but the main objection is that we don't see changes on the first hit. If 90% of files are similar - why to carry duplicated contents in the package build system?

v1.9.3-tplg2 is the first "topologies-only" release. This is new and I think what prompted this discussion.

On the other hand, re-releasing the same topologies again and again has always been routine. In fact I'd bet the majority of topologies rarely ever change from one release to the next (@plbossart, @ranj063 , confirm?). So yes, v1.9.3-tplg2 overwrites topology files most identical to v1.9.3. But v1.9.3 also overwrote topology files mostly identical to v1.9.2 before that and so forth and so on.

This is very different from firmware binaries (and dictionaries) which are almost always different from release to release (except in this -tplg2 case, which is why we they're not part of it)

There is no hard and fast rule but I think most people would intuitively expect some code changes from one release to the next while not being surprised by unchanged configuration files.

Could you just release new files or fixes then in a tar ball? The changes and updates will be visible from the first look.

That's an interesting idea but: 1) it would require much, much more release work and 2) it's not scalable if the number of "out of band" requests for brand new topologies grows. A complete release would have to be made out of an unknown number of tarballs.

There is a whole spectrum of possibilities (and complexity) between just one big archive with everything and releasing a single new or changed file at a time. We think drawing a line between topologies (which are basically product-specific config files) and the not-product-specific rest is the sweet spot.

(BTW I think most of us would like topology sources to be a different git repo eventually but we're still far from that yet. Maybe when we get to thesofproject/sof#3517?)

From the package perspective, it's more straight to add new files or update few files in the package and it helps to determine the testing requirements.

Aborted -tplg1 confusion aside (sorry about that), the churn in stable SOF branches is generally low and a quick look at the git log / changelog is the best to determine testing requirements: thesofproject/sof@v1.9.3...v1.9.3-tplg2

Note there may be other differences introduced not by SOF itself but by ALSA upgrades; hopefully none but potential counter-examples below. However I expect you to be the person the most informed and least confused by ALSA upgrades :-)

Also, it may be more handy to retain the updated paths - I speak about sof-tplg-v1.9.3 subtree in the v1.9.3 main package. The tar ball tree structure should be unchanged.

Let me see if I can fix that.

Also, there are no instructions (and may be script) to copy the files to the right location.

This is a flat archive with no directory structure, just a bunch of files all at the exact same level. I will add a one-line README stating where the files should be copied but I think a script that does the exact same thing as cp and nothing else would be overkill and do more harm than good; it would at the very least make this archive look more complicated than it really is.

@cc: @keqiaozhang , @kv2019i , @mengdonglin , @greg-intel

The problem with releasing code again is this:

either the code in the corresponding branch is exactly the same, then users get confused why the version number was incremented

or the code in the corresponding branch has some actual differences, then it requires a costly validation cycle again.

Forgot this: there's also the problem that the (time-consuming) signature process is not deterministic. It includes a date and also a salt in recent products. This will confuse users who try and fail to compare releases and do not know about

sof_ri_info: new 'reproducible.ri' feature + test sof#4889

A bash script which can show topology diffs from the previous full release:

PARENT=sof-tplg-v1.9.3

# update to parent
for i in *.tplg; do 
  if [ ! -r "${PARENT}/$i" ]; then
    echo "$i"
  else
    cmp "$i" "${PARENT}/$i"
    if [ $? -ne 0 ]; then
       echo "$i"
    fi
  fi
done

# parent to update
for i in ${PARENT}/*.tplg; do 
  b=$(basename "$i")
  if [ ! -r "$b" ]; then
    echo "Removed $b"
  fi
done

Result:

sof-adl-rt711-l2-rt1316-l01.tplg
sof-apl-nocodec-ci.tplg
sof-tgl-nocodec-ci.tplg

I assume that nocode-ci files are for your internal CI only, so I'll update our packages to add the extracted sof-adl-rt711-l2-rt1316-l01.tplg file.

Generating of the add-on (diff only) tar ball is really easy.

Also note that Fedora has SOF 2.0 packages where this file is missing!

The sof-adl-rt711-l2-rt1316-l01.tplg file is missing in the 2.1.1 release.

Could I (as a package maintainer) have a clear suggestion, what to do to support this hardware using v1.9.3 and v2.x firmware packages? I miss the continuity here.

@libinyang

The sof-adl-rt711-l2-rt1316-l01.tplg file is missing in the 2.1.1 release

After some git archeology I found that support for this topology was first added on the 21st of March 2022 in SOF commit 5ef9e496d046f3b. However stable-v2.1 had already been branched at that time (the 2.1 branch point / merge-base is https://github.com/thesofproject/sof/commits/980a5b25b3d97)

Because of a customer request, this topology was backported to v1.9.3 in https://github.com/thesofproject/sof/commits/acbd71eb9b07f72

@libinyang, @plbossart, @mengdonglin, should this topology be backported to stable-v2.1 too?

@libinyang

@libinyang, @plbossart, @mengdonglin, should this topology be backported to stable-v2.1 too?

Absolutely. We probably something to detect topology changes and backport them.

@libinyang

@libinyang, @plbossart, @mengdonglin, should this topology be backported to stable-v2.1 too?

Absolutely. We probably something to detect topology changes and backport them.

Ping. The package upgrade planning is not easy with this mismatch.

@libinyang

@libinyang ping

@perexg

@perexg @marc-hb @lgirdwood We'll backport the missing topology *.m4 for sof-adl-rt711-l2-rt1316-l01.tplg to stable-v2.1 Can we use tag v2.1a or v2.1a-tplg for this update? The previous suffixes -tplg1 and -tplg2 caused some confusion with topology source code v1 and v2.

For the binary release, may we only include new added sof-adl-rt711-l2-rt1316-l01.tplg this time, or still including all those unchanged *tplg as well？ Personally, I feel only including the added or updated *.tplg may be less error prone, but this really needs to be aligned with Redhat package upgrade plan.

@perexg

@perexg @lgirdwood @mengdonglin We now have v.2.1.1. Should we backport the patches to v2.1 or v2.1.1?

update: Mengdong told me that we didn't release v2.1 binaries. So it seems we should use v2.1.1.
Is it OK to you to use v2.1.1, @perexg @lrgirdwo

Yes, use the latest v2.1.1. Thanks.

Should we backport the patches to v2.1 or v2.1.1?

You cannot backport to a tag, tags don't move. You can only backport to a branch (stable-v2.1 in this case)

@libinyang

@libinyang Can you please share your SOF PR for stable-v2.1 to add this missing no-mic topology?

Done. thesofproject/sof#5870

@libinyang

Thank you, @libinyang

@perexg Now the missing topology has been merged by https://github.com/thesofproject/sof/commits/stable-v2.1
We plan to deliver a topology only release as v2.1.1a

Do you want the tar ball of v2.1.1a to including all topology binaries or only the missing topology binary?

Do you want the tar ball of v2.1.1a to including all topology binaries or only the missing topology binary?

The diff variant (missing and changed files only) is better for me.. Thank you.

@perexg

Do you want the tar ball of v2.1.1a to including all topology binaries or only the missing topology binary?

The diff variant (missing and changed files only) is better for me.. Thank you.

Thanks for your confirmation @perexg We'll prepare the v2.1.1a release with only the missing topology this week.

Ping.

Navigation Menu

Repository navigation

Partial, "topologies only" v1.9.3-tplg2 release #76

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Partial, "topologies only" v1.9.3-tplg2 release #76

Description

Activity

marc-hb commented on Apr 7, 2022

perexg commented on Apr 7, 2022

marc-hb commented on Apr 7, 2022

perexg commented on Apr 8, 2022

marc-hb commented on Apr 11, 2022

marc-hb commented on Apr 12, 2022

perexg commented on Apr 12, 2022

perexg commented on May 9, 2022

marc-hb commented on May 9, 2022

plbossart commented on May 10, 2022

perexg commented on May 27, 2022

lgirdwood commented on May 27, 2022

mengdonglin commented on May 30, 2022

libinyang commented on May 30, 2022

perexg commented on May 30, 2022

marc-hb commented on May 31, 2022

mengdonglin commented on May 31, 2022

libinyang commented on May 31, 2022

mengdonglin commented on Jun 10, 2022

perexg commented on Jun 13, 2022

mengdonglin commented on Jun 13, 2022

perexg commented on Jun 17, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions