Repository navigation
java.time.Instant obtained through Arrow Flight JDBC Driver is 16 hours different from the original timestamp #636
Description
Describe the bug, including details regarding any error messages, version, and platform.
java.time.Instantobtained through Arrow Flight JDBC Driver is 16 hours different from the original timestamp. This was originally discussed at Timestamps queried from InfluxDB 3 Core via JDBC Driver are inconsistent with those inserted influxdata/influxdb#25983 but it looks like there is mishandling within the Arrow Flight JDBC Driver.- I created a minimal reproducible unit test at https://github.com/linghengqian/influxdb-3-core-jdbc-test/blob/master/src/test/java/io/github/linghengqian/TimeDifferenceTest.java . The relevant unit tests use Influxdb 3 core .
- To execute it, just install
SDKMAN!andDocker CEin advance, then,
sdk install java 21.0.6-ms
git clone git@github.com:linghengqian/influxdb-3-core-jdbc-test.git
cd ./influxdb-3-core-jdbc-test/
sdk use java 21.0.6-ms
./mvnw -T 1C -Dtest=TimeDifferenceTest clean testClick me to view the core logic of the unit test🥯🥨🍟🧂🥖🥚🍔🦪🍜🍘
@Testcontainers
public class TimeDifferenceTest {
private final Instant magicTime = Instant.now().minusSeconds(10);
@Container
private final GenericContainer<?> container = new GenericContainer<>("quay.io/influxdb/influxdb3-core:911ba92ab4133e75fe2a420e16ed9cb4cf32196f")
.withCommand("serve --node-id local01 --object-store memory")
.withExposedPorts(8181);
@Test
void test() throws Exception {
try (InfluxDBClient client = InfluxDBClient.getInstance(
"http://" + container.getHost() + ":" + container.getMappedPort(8181),
null,
"mydb")) {
writeData(client);
queryDataByHttp();
queryDataByJdbcDriver();
}
}
private void writeData(InfluxDBClient client) {
Point point = Point.measurement("home")
.setTag("location", "London")
.setField("value", 30.01)
.setTimestamp(magicTime);
client.writePoint(point);
}
private void queryDataByHttp() throws URISyntaxException, IOException, InterruptedException {
URI uri = new URIBuilder().setScheme("http")
.setHost(container.getHost())
.setPort(container.getMappedPort(8181))
.setPath("/api/v3/query_sql")
.setParameter("db", "mydb")
.setParameter("q", "select time,location,value from home order by time desc limit 10")
.build();
HttpResponse<String> response = HttpClient.newHttpClient()
.send(HttpRequest.newBuilder().uri(uri).GET().build(), BodyHandlers.ofString());
assertThat(
new ObjectMapper().readTree(response.body()).get(0).get("time").asText(),
is(magicTime.atOffset(ZoneOffset.UTC).toLocalDateTime().toString())
);
}
private void queryDataByJdbcDriver() throws SQLException {
HikariConfig hikariConfig = new HikariConfig();
hikariConfig.setJdbcUrl("jdbc:arrow-flight-sql://" + container.getHost() + ":" + container.getMappedPort(8181) + "/?useEncryption=0&database=mydb");
try (HikariDataSource hikariDataSource = new HikariDataSource(hikariConfig);
Connection connection = hikariDataSource.getConnection()) {
ResultSet resultSet = connection.createStatement().executeQuery("select time,location,value from home order by time desc limit 10");
assertThat(resultSet.next(), is(true));
assertThat(resultSet.getString("location"), is("London"));
assertThat(resultSet.getString("value"), is("30.01"));
assertThat(resultSet.getTimestamp("time"), notNullValue());
// todo linghengqian why fail?
assertThat(resultSet.getTimestamp("time").toInstant(), is(magicTime));
}
}
}[INFO] Running io.github.linghengqian.TimeDifferenceTest
SLF4J(W): No SLF4J providers were found.
SLF4J(W): Defaulting to no-operation (NOP) logger implementation
SLF4J(W): See https://www.slf4j.org/codes.html#noProviders for further details.
2月 25, 2025 9:17:46 上午 org.apache.arrow.driver.jdbc.shaded.org.apache.arrow.memory.BaseAllocator <clinit>
信息: Debug mode disabled. Enable with the VM option -Darrow.memory.debug.allocator=true.
2月 25, 2025 9:17:46 上午 org.apache.arrow.driver.jdbc.shaded.org.apache.arrow.memory.DefaultAllocationManagerOption getDefaultAllocationManagerFactory
信息: allocation manager type not specified, using netty as the default type
2月 25, 2025 9:17:46 上午 org.apache.arrow.driver.jdbc.shaded.org.apache.arrow.memory.CheckAllocator reportResult
信息: Using DefaultAllocationManager at memory/netty/DefaultAllocationManagerFactory.class
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 3.520 s <<< FAILURE! -- in io.github.linghengqian.TimeDifferenceTest
[ERROR] io.github.linghengqian.TimeDifferenceTest.test -- Time elapsed: 3.457 s <<< FAILURE!
java.lang.AssertionError:
Expected: is <2025-02-25T01:17:34.640356152Z>
but: was <2025-02-24T09:17:34.640356152Z>
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:8)
at io.github.linghengqian.TimeDifferenceTest.queryDataByJdbcDriver(TimeDifferenceTest.java:89)
at io.github.linghengqian.TimeDifferenceTest.test(TimeDifferenceTest.java:50)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] TimeDifferenceTest.test:50->queryDataByJdbcDriver:89
Expected: is <2025-02-25T01:17:34.640356152Z>
but: was <2025-02-24T09:17:34.640356152Z>
[INFO]
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
- It is worth mentioning that https://github.com/linghengqian/influxdb-3-core-jdbc-test/blob/master/src/test/java/io/github/linghengqian/FlightSqlTest.java also does unit testing for the Arrow Flight Java API and does not have this problem.
- For the time zone of
Asia/Shanghai, the timestamp obtained by this unit test through the HTTP port is normal, but the timestamp obtained through the Arrow Flight JDBC Driver has a 16-hour error. This is unreasonable. - [JDBC] Timezone improvements for the JDBC driver #463 seems to suggest that use of
java.time.Instantneeds to be circumvented?
Activity
lidavidm commentedon Feb 25, 2025
Is this different than #463/#464?
linghengqian commentedon Feb 25, 2025
java.time.Instant.<absolute path to your arrow folder>is not fixed... Isn't the arrow git directory just the${user.dir}property in Maven?lidavidm commentedon Feb 25, 2025
You don't need the JNI build, just the pure Java build (so a simple
mvn install -DskipTestsshould be sufficient)lidavidm commentedon Feb 25, 2025
#463 lists
Instantas part ofjava.timeobjects that need to be better supported. More test cases demonstrating the problem would be helpful to have there, I think.lidavidm commentedon Feb 25, 2025
That said, yes, the driver's handling of timestamps has long been questionable, it's just a matter of defining what's "right", especially for supporting the legacy
java.sqldatetime types (which of course is not what you're concerned about here but is part of the problem).linghengqian commentedon Feb 25, 2025
-Dspotless.check.skip=trueparameter.java.time.Instantobtained from the query is 8 hours different from the original timestamp, which is still unreasonable.Click me to view the core logic of the unit test🥯🥨🍟🧂🥖🥚🍔🦪🍜🍘
org.apache.arrow:flight-sql-jdbc-driver:19.0.0-SNAPSHOT.lidavidm commentedon Feb 25, 2025
It would help to keep these observations in one spot. It's okay to comment in the draft PR, especially since you just tested it.
linghengqian commentedon Feb 25, 2025
lidavidm commentedon Apr 28, 2025
#463 was fixed by adding support for Instant et al, but I think the underlying correctness is still questionable, I have opened a new issue in #732 summarizing the observed behavior of PostgreSQL's JDBC driver. I agree that the Flight SQL driver is not handling things correctly.