Skip to content

Inventory ID update - #417

Draft
varunghat wants to merge 18 commits into
phoible:masterfrom
varunghat:feat/update-inventoryID
Draft

varunghat wants to merge 18 commits into
phoible:masterfrom
varunghat:feat/update-inventoryID

Conversation

@varunghat

Copy link
Copy Markdown
Contributor

Adds generate-inventory-ids.R and sync-inventory-ids.R.
Updates all references to the current IDs (called Legacy here) to new IDs in mapping files, raw-data, phoible.csv
Updates tests to read inventoryIDs as characters, hardcoded legacy IDs changed to new IDs.
Modified aggregate-raw-data to add a method of retiring inventories through soft delete

  1. generate-inventory-ids.R - New IDs are character strings of the format "pXXXXX" where XXXXX is a 5 digit hexadecimal number generated through multiplicative hashing. This ensures deterministic ID generation (so no random generation and retry issues) and it is non-sequential.
    The p (for phoible) is to ensure the new IDs throw an error if any old code reads it as an integer, and 5 digits to differentiate it from the 4 digit ID before. This should reduce instances of people confusing the new IDs for an older integer ID due to similarity (eg, p1200 could be mistaken to be same as 1200 from the old IDs)

  2. sync-inventory-ids.R - Script to go through each source and generate and bake in hex InventoryIDs. Code exists so that we can add new inventories easily without manually having to bake in the InventoryIDs like before.

  3. Soft delete inventories - mapping/InventoryID-Retired.csv can be used to retire inventories without modifying them in raw-data. Declaring an ID in the csv will automatically exclude it from the final phoible.csv in aggregate-raw-data.R

  4. New mapping files - InventoryID-Legacy.csv maps the current (Legacy) IDs to the new hex IDs. Rest of the mapping files now use the new IDs.

  5. Tests - tests updated to read IDs as character strings. Hardcoded entries of integer IDs updated to the hex IDs.

@drammock Let me know if there's any issue with any of the above parts.

NOTE: Will need to create an issue on the CLDF side to redirect pages with old IDs to the newer ones.

@github-actions

Copy link
Copy Markdown

phonemes:
Phonemes: .
Features: ..1

══ Failed ══════════════════════════════════════════════════════════════════════
── 1. Failure ('test-phonemes.R:140:5'): within inventories, phonemes have disti
PHONEMES WITH INDISTINCT FEATURE VECTORS:

A tibble: 50 × 2

Phonemes InventoryIDs

1 c̤ ɟ̤ p09343 p2b310
2 s s̺ p1976c p9067e pb4387
3 z z̺ p1976c p9067e
4 ə ɜ p1a049 p288c4 p4a891 p4ce33 p62fd1 p67e31 p6f438 p7f861 pa597e pac5…
5 s̺̠ ʃ p1ae70
6 b̥ p p205ad p2a717 p546df p63f86 p97969 pa3c42 pa77d1 pae36d pbdc14 pcc0…
7 d̥ t p205ad pdb1b4
8 d̥ʑ̥ tɕ p205ad p70820
9 ɡ̊ k p205ad p2a717 p546df p63f86 pa3c42 pa77d1 pae36d pb04dc pbdc14 pcc0…
10 e̞ ɛ̝ p205ad
11 a ä p2215b p6d13b
12 d̥ʒ̥ t̠ʃ p2a717 pa3c42 pae36d pbdc14 pcc05c pe88ec pfe5e0
13 d̺̥ t̺ p2a717 p546df p63f86 pa3c42 pa77d1 pae36d pbdc14 pcc05c pe88ec pef4…
14 d̺̥z̺̥ t̺s̺ p2a717 p63f86 pbdc14 pcc05c pe88ec pef488 pfe5e0
15 ɡ̊ʷ kʷ p2a717 p546df p63f86 pa3c42 pa77d1 pae36d pcc05c pe88ec pef488 pfe5…
16 b̰ p̰ p2e8de p97af7
17 ë ë̞ p375e9 p6d13b
18 d̠ʒʰ t̠ʃʰ p3c449
19 kǀ͓ kǀ͓x p63ce1
20 kǂ͓ˡ kǂ͓ˡx p63ce1
21 ŋ̥ǃˠˀ ŋ̥ǃˀ p63ce1
22 ö ö̞ p6d13b
23 b̥ː pː p70820
24 d̥z̥ ts p70820
25 d̥z̥ʲ tsʲ p70820
26 d̥ː tː p70820
27 ɡ̊ː kː p70820
28 ɲc ɲcʲ p84c82
29 eu̯ e̞u̯ p9067e
30 oi̯ o̞i̯ p9067e
31 d̪̥ t̪ p97969
32 b̤̥ pʰ p98cb1
33 d̤̥ tʰ p98cb1
34 ɡ̤̥ kʰ p98cb1
35 ə̃ ɜ̃ pa597e
36 d̪z̪ d̻z̻ pa950d
37 s̪ s̻ pa950d
38 t̪s̪ t̻s̻ pa950d
39 z̪ z̻ pa950d
40 n n̺ pb4387
41 ɯ ɯ̝ pc4061
42 f v̥ pdb1b4
43 s z̥ pdb1b4
44 ɣ̊ x pdb1b4
45 ʃ ʒ̊ pdb1b4
46 ɐ̆ ɐ̹̆ peaf05
47 əː ɜː pef7a4
48 aː äː pf3eb5
49 i̯œ i̯œ̞ pf9ecf
50 d̪ d̻ pfeebd

══ DONE ════════════════════════════════════════════════════════════════════════

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant