You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Adds generate-inventory-ids.R and sync-inventory-ids.R.
Updates all references to the current IDs (called Legacy here) to new IDs in mapping files, raw-data, phoible.csv
Updates tests to read inventoryIDs as characters, hardcoded legacy IDs changed to new IDs.
Modified aggregate-raw-data to add a method of retiring inventories through soft delete
generate-inventory-ids.R - New IDs are character strings of the format "pXXXXX" where XXXXX is a 5 digit hexadecimal number generated through multiplicative hashing. This ensures deterministic ID generation (so no random generation and retry issues) and it is non-sequential.
The p (for phoible) is to ensure the new IDs throw an error if any old code reads it as an integer, and 5 digits to differentiate it from the 4 digit ID before. This should reduce instances of people confusing the new IDs for an older integer ID due to similarity (eg, p1200 could be mistaken to be same as 1200 from the old IDs)
sync-inventory-ids.R - Script to go through each source and generate and bake in hex InventoryIDs. Code exists so that we can add new inventories easily without manually having to bake in the InventoryIDs like before.
Soft delete inventories - mapping/InventoryID-Retired.csv can be used to retire inventories without modifying them in raw-data. Declaring an ID in the csv will automatically exclude it from the final phoible.csv in aggregate-raw-data.R
New mapping files - InventoryID-Legacy.csv maps the current (Legacy) IDs to the new hex IDs. Rest of the mapping files now use the new IDs.
Tests - tests updated to read IDs as character strings. Hardcoded entries of integer IDs updated to the hex IDs.
@drammock Let me know if there's any issue with any of the above parts.
NOTE: Will need to create an issue on the CLDF side to redirect pages with old IDs to the newer ones.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds generate-inventory-ids.R and sync-inventory-ids.R.
Updates all references to the current IDs (called Legacy here) to new IDs in mapping files, raw-data, phoible.csv
Updates tests to read inventoryIDs as characters, hardcoded legacy IDs changed to new IDs.
Modified aggregate-raw-data to add a method of retiring inventories through soft delete
generate-inventory-ids.R - New IDs are character strings of the format "pXXXXX" where XXXXX is a 5 digit hexadecimal number generated through multiplicative hashing. This ensures deterministic ID generation (so no random generation and retry issues) and it is non-sequential.
The p (for phoible) is to ensure the new IDs throw an error if any old code reads it as an integer, and 5 digits to differentiate it from the 4 digit ID before. This should reduce instances of people confusing the new IDs for an older integer ID due to similarity (eg, p1200 could be mistaken to be same as 1200 from the old IDs)
sync-inventory-ids.R - Script to go through each source and generate and bake in hex InventoryIDs. Code exists so that we can add new inventories easily without manually having to bake in the InventoryIDs like before.
Soft delete inventories - mapping/InventoryID-Retired.csv can be used to retire inventories without modifying them in raw-data. Declaring an ID in the csv will automatically exclude it from the final phoible.csv in aggregate-raw-data.R
New mapping files - InventoryID-Legacy.csv maps the current (Legacy) IDs to the new hex IDs. Rest of the mapping files now use the new IDs.
Tests - tests updated to read IDs as character strings. Hardcoded entries of integer IDs updated to the hex IDs.
@drammock Let me know if there's any issue with any of the above parts.
NOTE: Will need to create an issue on the CLDF side to redirect pages with old IDs to the newer ones.