You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We use txn client to scan data, with a certain start key and end key, but it does not woking with a KV:Storage:InvalidReqRange error, here is the log output by Tikv
Could you offer a minimal step to reproduce this issue?
We don't know what that error means, so we haven't been able to reproduce that issue with a small amount of data.
Only when loading LDBC SF100 data in our encoding will this problem occur in some specific scan operations.
The details of the problem we encountered here are as follows
Load the data of sf100 to TIKV in our encoding method
Rewritten RegionStoreClient in TIKV java and added the function to send scan request according to star, end, limit and version.
public List<KvPair> scan(
BackOffer backOffer, ByteString startKey,ByteString endKey, int limit, long version, boolean keyOnly) {
boolean forWrite = false;
while (true) {
// we should refresh region
region = regionManager.getRegionByKey(startKey, backOffer);
Supplier<ScanRequest> request =
() ->
ScanRequest.newBuilder()
.setContext(
makeContext(
getResolvedLocks(version), this.storeType, backOffer.getSlowLog()))
.setStartKey(codec.encodeKey(startKey))
.setVersion(version)
.setKeyOnly(keyOnly)
//.setEndKey(codec.encodeKey(endKey)) //comment this code,KV:Storage:InvalidReqRange will not happen
.setLimit(Math.min(limit, conf.getScanBatchSize()))
.build();
Query a prefix such as "2, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 51, -36, 2, 0, 0, 26, 0, 0, 1 , 19, 0, 0”.And with this prefix ,
star will be "2, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 51, -36, 2, 0, 0, 26, 0, 0, 1, 19, 0, 0",
end will be "2, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 51, -36, 2, 0, 0, 26, 0, 0, 1, 19 , 0, 1” ,
Send scanRequest, you will encounter the "KV:Storage:InvalidReqRange" on TIKV's log mentioned earlier.
Not all scan will encounter such problems, only some specific prefixes will encounter such problems, but we cannot find any difference between these prefixes and other normal prefixes.
So we want to know what this error really means, and see if we can find what the wrong prefix and data have in common with it.
Could you offer a minimal step to reproduce this issue?
...
So we want to know what this error really means, and see if we can find what the wrong prefix and data have in common with it.
We did not encounter this problem with small amounts of data.
The cause of error KV:Storage:InvalidReqRange is that end is bigger than upper_bound(i.e., Some([2, 0, 0, 0, 0, 0, 0, 1, 255, 1, 0, 0, 0, 51, 220, 2, 0, 255, 0, 26, 0, 0, 1, 19, 0, 1, 255, 0, 0, 0, 0, 0, 0, 0, 0, 247]) >Some([2, 0, 0, 0, 0, 0, 0, 1, 255, 1, 0, 0, 0, 51, 220, 2, 0, 255, 0, 26, 0, 0, 1, 19, 0, 0, 255, 0, 0, 0, 0, 0, 0, 0, 0, 255, 1, 0, 0, 0, 67, 139, 1, 0, 255, 0, 30, 0, 0, 0, 0, 0, 0, 255, 0, 0, 0, 0, 0, 0, 0, 0, 247]), and out of the region scope.
To address this issue, I think you can assign the end as the smaller one between start+1 and upper_bound of the region (RegionStoreClient.region.getEndKey())
Thanks!
After adding the judgment of the endkey of the region, the new scan function can be executed correctly.
The reason why this new scan function is defined is that the “limit” of most of our scan request will be relatively large, but the results of many queries may not be very large. However, the original scan function that supports “version” does not support sinking the endkey to the TIKV server. It only pulls the limit (or scanBatchSize) amount of data to the client and then filters it in the iterator, which is too inefficient for us. .
So we want to use the endkey in the interface, and then only the data we need will be transmitted when querying.
Activity
iosmanthus commentedon Oct 19, 2022
Could you offer a minimal step to reproduce this issue?
AIFun commentedon Oct 19, 2022
We don't know what that error means, so we haven't been able to reproduce that issue with a small amount of data.
Only when loading LDBC SF100 data in our encoding will this problem occur in some specific scan operations.
The details of the problem we encountered here are as follows
star will be "2, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 51, -36, 2, 0, 0, 26, 0, 0, 1, 19, 0, 0",
end will be "2, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 51, -36, 2, 0, 0, 26, 0, 0, 1, 19 , 0, 1” ,
Send scanRequest, you will encounter the "KV:Storage:InvalidReqRange" on TIKV's log mentioned earlier.
So we want to know what this error really means, and see if we can find what the wrong prefix and data have in common with it.
AIFun commentedon Oct 19, 2022
We did not encounter this problem with small amounts of data.
pingyu commentedon Oct 19, 2022
The cause of error
KV:Storage:InvalidReqRangeis thatendis bigger thanupper_bound(i.e.,Some([2, 0, 0, 0, 0, 0, 0, 1, 255, 1, 0, 0, 0, 51, 220, 2, 0, 255, 0, 26, 0, 0, 1, 19, 0, 1, 255, 0, 0, 0, 0, 0, 0, 0, 0, 247])>Some([2, 0, 0, 0, 0, 0, 0, 1, 255, 1, 0, 0, 0, 51, 220, 2, 0, 255, 0, 26, 0, 0, 1, 19, 0, 0, 255, 0, 0, 0, 0, 0, 0, 0, 0, 255, 1, 0, 0, 0, 67, 139, 1, 0, 255, 0, 30, 0, 0, 0, 0, 0, 0, 255, 0, 0, 0, 0, 0, 0, 0, 0, 247]), and out of the region scope.To address this issue, I think you can assign the
endas the smaller one betweenstart+1andupper_boundof the region (RegionStoreClient.region.getEndKey())What's the usage of the new
scanmethod ?AIFun commentedon Oct 19, 2022
Thanks!
After adding the judgment of the endkey of the region, the new scan function can be executed correctly.
The reason why this new scan function is defined is that the “limit” of most of our scan request will be relatively large, but the results of many queries may not be very large. However, the original scan function that supports “version” does not support sinking the endkey to the TIKV server. It only pulls the limit (or scanBatchSize) amount of data to the client and then filters it in the iterator, which is too inefficient for us. .
So we want to use the endkey in the interface, and then only the data we need will be transmitted when querying.
The new code is
pingyu commentedon Oct 19, 2022
Got it.
I think this change is great, and it's very welcome to raise a pull request.
@iosmanthus How do you think ?
iosmanthus commentedon Oct 20, 2022
@AIFun feel free to open a pull request, we could use this scan to implement all variant versions of the scan.
AIFun commentedon Oct 31, 2022
There are still some bug in my code. I'm trying to fix them.I'll open a PR after it work.
github-actions commentedon Dec 1, 2022
This issue is stale because it has been open 30 days with no activity.
txn: fix scan missing endKey in V2 API