Not Displaying G4 Share Receiver data directly

So based on Gitter chats, it seems this issue is only affecting a small number of us, but nonetheless, we exist out there.

For some reason, Loop won't display glucose data from G4 Share Receiver, even though the Share 2 app is getting it just fine. While I can hook up Dexcom Share account and get cloud data by having Share 2 app send data to the cloud, then back down to Loop, it's not as reliable and I'd prefer to be able to get the glucose data directly without the cloud involved.

Some environmental factors:
-Mac OSX 10.12.1 Sierra
-Xcode 8.1
-Loop 1.1 Dev branch
-iPhone 6s
-iOS 10.1

Reports/Logs:
Issue Report in Loop: https://gist.github.com/ruess/4ada2dee7403c04165dd7be1408a7eef
mLab Log: Errors: https://gist.github.com/ruess/8bcc150f482b0ca49bc0c8287b38264c
mLab Events: https://gist.github.com/ruess/d94ae2355246b33a71558fb1078e40c8
Xcode Loop Output: https://gist.github.com/ruess/384dc04dccdf9a3d37644d9416cc925d

Screencast of Issue:
https://www.youtube.com/watch?v=VVK2JZcn1BY&feature=youtu.be
(note: time in video at top of iPhone screen is incorrect: should be 6:01 PM (apple thing) - off about 3 hours from last glucose reading which came from Dexcom cloud. Also, while many settings are blank, I've previously tried with all settings filled and it made no difference)

My Hypotheses:
-Maybe my G4 Platinum with Share receiver is funky somehow - I picked it up off eBay - it may have been refurbished. Manufacture date: 2105-07-07. If for some reason, it's not matching what Loop expects as a device, perhaps it's throwing its data out. There's nothing that indicates that it should be though, as it's working flawlessly with Share 2 App (dexcom share site, nightscout, etc.)
-I'm doing something wrong!

This is pretty specific device info in DeviceDataManager.swift (line 598):

`// "Dexcom G4 Platinum Transmitter (Retail) US" - see https://accessgudid.nlm.nih.gov/devices/search?query=dexcom+g4

let device = HKDevice(name: "G4ShareSpy", manufacturer: "Dexcom", model: "G4 Share", hardwareVersion: nil, firmwareVersion: nil, softwareVersion: String(G4ShareSpyVersionNumber), localIdentifier: nil, udiDeviceIdentifier: "40386270000048")`

Just wondering what happens if there's any consequences for a device not matching these exact specifications in terms of whether to keep or throw the data? (and I am speaking out of total ignorance here)

Thanks for starting this thread and providing the detailed logs!

Is your issue intermittent, or has Loop never been able to read from your receiver when the G4 Share Receiver switch is toggled?

It's worth noting that the Share2 app must already be running when the switch is toggled, since the switch causes it to check for an already-paired receiver rather than scan for one. If the Share2 app ever stops, the switch may need to be toggled again. I know this doesn't apply to the screencast you shared. Generally I need to toggle the switch only when my phone restarts.

Based on the Bluetooth events in your g4 logs ("mLab Events"), Loop found your G4 receiver and subscribed to changes in its rx characteristic, which should show all the receiver's responses as it interacts with the Share2 app.

If your problem is that the receiver remains connected for a while then disconnects, see if any didDisconnectPeripheral events appear in that log later with more information.
If your problem is a consistent inability to get glucose from G4, the next questions are whether the rx characteristic is indeed updating (maybe your receiver uses a different characteristic?) and whether its protocol is somehow different. The easiest way to make changes which can answer those (I think) is for you to drag the source files for G4ShareSpy into Loop, and remove the compiled version from Frameworks/ and from the "Link Binary With Libraries" phase under "Build Phases" for the Loop target. Then you can make changes to G4ShareSpy which will be included in your build of Loop. In the didUpdateValueFor callback, add some logging, e.g.:

    func peripheral(_ peripheral: CBPeripheral, didUpdateValueFor characteristic: CBCharacteristic, error: Error?) {
+       NSLog("bytes from receiver: \(characteristic.value)")
        if let error = error {
            delegate?.bluetoothManager(self, didError: error)
            return
        }
        if let value = characteristic.value {
            self.delegate?.bluetoothManager(self, didReceiveBytes: value)
        }
}

Another way to sanity-check that your receiver works the same as G4ShareSpy expects is to check what services/characteristics it offers, using an app like LightBlue. They should match these, which are taken from here. If nothing comes of the above, it may be worth notifying on all your receiver's characteristics to see what the Share2 app is doing with it.

Let me know what you find!

(That line you pasted from DeviceDataManager affects only the metadata which Loop adds to glucose values when it stores them in HealthKit, which it does only temporarily until Dexcom's app adds them. Loop finds the G4 receiver by looking for an already-connected peripheral whose services match a particular UUID.)

Hi,

To add some answers as I am one of the other people having this issue.

This is a constant issue for most of us. We (myself included) have never been able to get data directly from the receiver while other methods all work perfectly.
My Receiver is brand new
I am intending to rebuilt a OpenAPS rig and will test to use openxshareble to see if it works there.

Thank you all for looking into this. I have G4+share and I've had no issues with Loop getting bg data directly on various Loop versions over the last couple of months, up to version 1.0 now compiled with Xcode 8.0 and installed on an iPhone6s running 10.0.2. I recently tried to help a friend install Loop and everything went well except we hit the G4+share problem: his Loop (first 1.0, now 1.1) has never been able to get bg data directly. I've checked my and his Dex receivers and they happen to be exactly the same, both obtained directly from Dexcom in 2015:

Part#: MT22495
Part Rev: 11
SW #: SW10586
SW Rev: 3.0.1.026

His Share2 consistently works fine, and is sharing his bg data to Dex share server (which Loop is able to get when the "G4 Share Receiver" button is off). We've tried phone power up/down, reinstalling Loop, reinstalling Share2, bluetooth on/off, "G4 Share Receiver" button on/off. No luck.

@dm61

@dm61, can you try your receiver with your friend's phone, and his with yours? Wondering if it's an issue with the phone and not the receiver.

@mddub

@mddub , good idea but no, we do not live in the same location, swapping receivers would not be practical.

@mddub

Hey @mddub,

Great and thorough response -thanks for that and for the guidance on next steps. Did a quick check with LightBlue to see what my device read versus the reference you linked and came out with the following.

The only one that wasn't exactly the same was the DeviceInfo UUID, which I'm assuming is because each device has a different one?

 Match YES = case CGMService = "F0ACA0B1-EBFA-F96F-28DA-076C35A521DB"
Match NO =  case DeviceInfo = "00001804-0000-1000-8000-00805f9b34fb" 
      Mine:  Labeled 'Peripheral UUID' = "59025C93-6FA2-A77A-D5CD-E811694A8550"

Match YES =    case Tx = "F0ACB20A-EBFA-F96F-28DA-076C35A521DB"
Match YES  =   case Rx = "F0ACB20B-EBFA-F96F-28DA-076C35A521DB"
Match YES =    case Auth = "F0ACACAC-EBFA-F96F-28DA-076C35A521DB"
Match Yes =    case Heartbeat = "F0AC2B18-EBFA-F96F-28DA-076C35A521DB"
Match Yes =    case ModelNumberString = "2A24"
Match Yes =    case FirmwareRevisionString = "2A26"
Match Yes =    case HardwareRevisionString = "2A27"
Match Yes =    case ManufacturerNameString = "2A29"

Let me know if all above seems ok. I'm working on integrating a custom logging G4SpyShare into Loop and will update when that's up and logging. Mobile development is brand new to me.

Thanks!

That list looks good to me. I was speculating that maybe, in the event that you find there's nothing happening on the rx characteristic, there could be activity on one of the characteristics found by the xDrip folks which Loop doesn't currently listen to, or on a different characteristic you find not documented there. Worth exploring after you've looked into what bytes come in on the expected rx characteristic.

I'm hopeful that you'll be able to find something by modifying G4ShareSpy to monitor what the receiver does. Based on your report, I have no good guess as to what's different about your receiver (or phone?!).

@walker0

Hmm - so far not finding too much, though as @walker0 mentioned, the fact that no specific bytes are being listed in the log may mean that data is being received may not be understood.

Here's the gist:
https://gist.github.com/ruess/76eea695e83bceb3cfbaf961273f5403

Any ideas for further testing?

Thanks!

No, this is great! The problem is that Data in Swift 3 doesn't print its byte values like NSData did, which is why those Optional(...) strings are so unhelpful. Try this instead:

    func peripheral(_ peripheral: CBPeripheral, didUpdateValueFor characteristic: CBCharacteristic, error: Error?) {
        if let error = error {
            delegate?.bluetoothManager(self, didError: error)
            return
        }
        if let value = characteristic.value {
+           let hexString = NSMutableString(capacity: value.count * 2)
+           for byte in value {
+               hexString.appendFormat("%02x", byte)
+           }
+           NSLog("bytes from receiver: \(hexString)")
            self.delegate?.bluetoothManager(self, didReceiveBytes: value)
        }
    }

And while you're at it, in Receiver.swift:

    private func parseMessage(_ message: Data, receivedAt: Date) {
+       NSLog("parsing")
        if let systemTimeMessage = SystemTimeMessage(data: message) {
+           NSLog("got system time message: \(systemTimeMessage)")
            clockOffset = receivedAt.timeIntervalSince1970 - Double(systemTimeMessage.time)
            if let pending = glucoseHistoryAwaitingClock {
                emitGlucose(pending, clockOffset: clockOffset!)
                glucoseHistoryAwaitingClock = nil
            }
        } else if let glucoseHistoryMessage = GlucoseHistoryMessage(data: message) {
+           NSLog("got glucose history message: \(glucoseHistoryMessage)")
            if let offset = clockOffset {
                emitGlucose(glucoseHistoryMessage, clockOffset: offset)
            } else {
                glucoseHistoryAwaitingClock = glucoseHistoryMessage
            }
        }
    }

@mddub

Thanks @mddub - this is great. I've been trying to implement this code, but having trouble with it showing literally nothing most of the time. For the last round it was doing the same and every 3 or 4 build attempts it would suddenly just start logging stuff randomly. Any suggestions on how I could force it to try listening to G4?

Thanks!

@walker0

Thanks to @walker0, who kindly helped add the new lines of code, I have some more interesting data to look at I hope:

https://gist.github.com/ruess/d2ddcfe6d3ed6194a1c6b770c4c18cb1

Lines 79, 142, and 205 seem to have some glucose readings (perhaps raw?)

Super helpful! Thanks. I have a suspicion of what's going on, related to this line. From those logs, Loop is picking up the glucose history fine, but it's never picking up a valid system time message (which is needed to determine the receiver's time offset and assign absolute timestamps to the records).

Is your receiver's time set correctly? If not, what's it set to/how much is it off by?

I notice that the first record in the glucose history message has a display time of 2016-09-02 04:32:56 +0000. This is computed by adding the "Dexcom clock start" (2009-01-01) to the system time seconds in the glucose record.

@mddub

Wow nice catch @mddub! I'm embarrassed that while the actual time (in hours) and the day of the month was correct on the Dexcom Receiver, the month was 2 months behind! When I got it from ebay, I'm must have looked and simply updated the time, assuming everything else was right -bad assumption.

Anyway, I've updated the time, day, and month on Dexcom now, but still no dice. Even tried restarting Loop, Share 2 apps, as well as re-pairing the Dexcom with Share 2, but still not working.

I've done a new log test with the updated time and am getting some interesting things.

For instance, on this line from the new log:
2016-11-02 10:03:23.411592 Loop[1064:692004] got glucose history message: GlucoseHistoryMessage(records: [G4ShareSpy.GlucoseHistoryRecord(sequence: 1824, systemTime: 241569970, wallTime: 2016-11-02 17:03:55 +0000, glucose: 85, isDisplayOnly: false, trend: 4)
. . . we see that the time it was logged was at 10:03 AM my time (GMT -5), but system time is: 241569970 which if I assume is in epoch = Sat, 27 Aug 1977 22:46:10 GMT (scratching my head there - unless that's measuring the time from 2009 as you mentioned in your last comment, instead of January 1, 1970)
. . . also, the wallTime, which I assume is the Dexcom's time at reading was supposedly today at 5:03 PM GMT +0000 which is my time at 13:03:55 Wednesday November 2, 2016. Yet of course, when I took this log, it was actually around 10 AM, so why would it be off over 3 hours?

Just for reference now, I'm including a screenshot of my current time (now it's almost 11, still a few hours before 1:03 PM).

Also, here's the latest log:
https://gist.github.com/ruess/ff76a2e25e11e644735717a6360696f9

Hoping we're getting real warm now!

Just for reference and as a second point, I can confirm that Loop/Phone, pump and Dexcom time are all the same. Max out by a minute or two from each other.

@ruess

Same here. I had my friend check the time settings on his receiver and his pump, and both were within a minute from the Phone.

@ruess , @sjolundjohn , have you tried to hard reset your receivers? (with a pin). It has just occurred to me that's one thing I've not tried.

@dm61

Hey @dm61 - I did do a hard reset with a pin when I first got the receiver, but not since then and not since after attempting Loop. However, since there are readings coming to Loop, I'm interested to see why these are not being displayed in Loop. Maybe I can try a hard reset if we still haven't figured out by the time I need to do a sensor change so as to not lose 2 hours :) (unless we come to the consensus that it's needed sooner)

@dm61

The Dexcom seconds aren't epoch time, but rather seconds since 2009-01-01, so 241569970 is 2016-08-27 22:46:10. That timestamp may have been saved on the record when it was generated, though, so it doesn't reflect your change to the receiver wall time. You can see how G4ShareSpy translates the bytes into those fields here.

The glucose history pages come in date-ascending order, so those first readings in the list should be a few hours old. Is Xcode/Gitter truncating those long GlucoseHistoryMessage lines, or can you include them in full, up to the most recent readings? Can you let it run for several 5-minute cycles and paste the logs from that?

I can think of a couple of ways to fix this problem, but I could use more data to make sure it will work consistently (and in a bad coincidence of timing, my G4 transmitter died this week, so I'm on G5 now and need your logs).

What G4ShareSpy does is passively listen to messages from the receiver as it interacts with the Share2 app. There are only two types of messages that matter: glucose history and system time. A glucose history message is easy to identify: a 4-byte message header, a 28-byte glucose history header, then many 13-byte glucose records. A system time message, on the other hand, has nothing to identify it except that it has a 4-byte payload representing the current "Dexcom system time." In order to avoid misidentifying some other 4-byte message as system time (and thus misinterpreting the time for glucose records), I chose an arbitrary threshold of some time within 7 days of today. This is why the system time message from your receiver is being ignored. Because the system time on the glucose records can't be reconciled, they are never released to Loop.

If those new logs were generated after you changed the clock, then it looks like the Dexcom system time didn't change after you fixed the date:

2016-11-02 10:03:33.080227 Loop[1064:692366] bytes from receiver: 010a0001d91e660e985d

010a0001 = [1, 10, 1] (message header)
d91e660e = 241573593 (system time) = 2016-08-27 23:46:33
985d = CRC

Would you mind resetting the receiver with a paperclip as @dm61 suggested and pasting logs from a few 5-minute cycles to see if the time you set after restart becomes the new system time?

@mddub

Hey @mddub,

Thanks for the info above. I wish I could say I understood it all, but still having trouble wrapping my head around everything related to the time.

I've done a much more extensive log and I can say that, no, nothing is being truncated from the Xcode log from what I can tell - though on the gist, you may need to sidescroll certain lines to see the entire thing.

The latest log shows the system time of the dexcom as stubbornly remaining around the value of 241626668 (which I convert to 2016-08-28 22:31:08), and other values around that amount - basically all of them after conversion falling onto 2016-08-28. This is after multiple attempts to hard reset the device. I don't know if I'm doing it wrong, but I've followed the Dexcom user guide's instructions of having the device plugged while pushing the back with a paperclip for over 15 seconds. The device vibrates and restarts, but my recent BG history, the transmitter ID, and the recent time I set it to all remain - nothing gets wiped out. I tried with and without it being plugged in, but to no avail. Not sure where it got 2016-08-28 and why it's sticking to that - but it's strange indeed.

Another bizarre thing to me that I still can't understand is the other timestamp of "walltime." For instance, a reading that just came through the log is:
2016-11-03 8:47:09 +0000, which as you can see from the image, has not even occurred yet at the time it was reported and at the time the screenshot was taken. It's basically a time about 3 hours in the future.

Either way, here's the log:
https://gist.github.com/ruess/14c06d06cac9a1c7b1e893cde5f57384

Thanks much!

Yeah, the +0000 there is misleading; it's not actually part of the wall time, since wall time is unzoned. I have no idea why it's showing a future time. Anyway, the wall time doesn't matter.

There are two times on the receiver: "wall time", which is what you see on the device, and "system time", which is generally not visible to/modifiable by the user, but can be used to determine the absolute time of a reading.

System time is the only one that matters for parsing glucose history. We need the system time at which a reading occurred, and at some other point, a report of the receiver's current system time. It really doesn't matter what the system time is -- it could be off by years -- because we just need to know its offset to know the exact time of the readings.

Too bad you can't do a hard reset to change your system time. Lesson learned! I think the best solution is the simplest: expand the tolerance of valid system times. Let's try allowing anything within 10 years of today.

Can you update your Cartfile as below, run carthage update, and confirm that fixes it? If so, I'll merge the change.

 github "LoopKit/xDripG5" ~> 0.8.0
 github "i-schuetz/SwiftCharts" ~> 0.5.0
 github "mddub/dexcom-share-client-swift" ~> 0.2.0
-github "mddub/G4ShareSpy" ~> 0.3.0
+github "mddub/G4ShareSpy" "less-strict-system-time"
 github "ps2/rileylink_ios" ~> 0.12.6
 github "amplitude/Amplitude-iOS" ~> 3.8.5

@mddub

Holy Crap!! I nearly fell off my chair when I saw the glucose change without a little cloud icon!

Updated carthage and we're good to go - you're awesome @mddub!

@mddub

Wow, fantastic, great work @mddub and @ruess. Can the solution be merged into master, or should this workaround only be done in exceptional cases when the receiver system time is way off for some reason?

@ruess

@ruess: 🎉🎉🎉

@dm61: I feel comfortable merging the fix into master. (Well, dev first, to get a couple more confirmations.)

Confirmed that this solved my problem - fantastic!

It also fixed it for me 👍 💯 yay!

@mddub

Looks good - closing for the time being as I think we've demonstrated the fix works for more than just myself.

Thanks again for your help and guidance with troubleshooting @mddub!

Great! For anyone else who finds this issue: the fix is in dev now (no need to change any files or run carthage update), will be part of whatever the next release after v1.1 is.

Navigation Menu

Repository navigation

Not Displaying G4 Share Receiver data directly #242

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Not Displaying G4 Share Receiver data directly #242

Description

Activity

ruess commented on Oct 31, 2016

mddub commented on Oct 31, 2016

sjolundjohn commented on Oct 31, 2016

dm61 commented on Oct 31, 2016

mddub commented on Oct 31, 2016

dm61 commented on Oct 31, 2016

ruess commented on Oct 31, 2016

mddub commented on Nov 1, 2016

ruess commented on Nov 1, 2016

mddub commented on Nov 1, 2016

ruess commented on Nov 1, 2016

ruess commented on Nov 2, 2016

mddub commented on Nov 2, 2016

ruess commented on Nov 2, 2016

sjolundjohn commented on Nov 2, 2016

dm61 commented on Nov 2, 2016

ruess commented on Nov 2, 2016

mddub commented on Nov 2, 2016

ruess commented on Nov 3, 2016

mddub commented on Nov 3, 2016

ruess commented on Nov 3, 2016

dm61 commented on Nov 3, 2016

mddub commented on Nov 4, 2016

sjolundjohn commented on Nov 4, 2016

ianjorgensen commented on Nov 4, 2016

ruess commented on Nov 4, 2016

mddub commented on Nov 4, 2016

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions