Skip to content

update starrocks result to 3.0.0-preview - #72

Merged
nickitat merged 2 commits into
ClickHouse:mainfrom
silverbullet233:3.0-preview
Feb 14, 2023
Merged

nickitat merged 2 commits into
ClickHouse:mainfrom
silverbullet233:3.0-preview

Conversation

@silverbullet233

@silverbullet233 silverbullet233 commented Feb 10, 2023 •

Copy link
Copy Markdown
Contributor

The main changes in this PR include:

  1. update the version of starrocks to 3.0.0-preview and adjust the way of data import.
  2. Fix Q28, we did not handle the \ character correctly and this mistake didn't be realized by us before, sorry for that.
  3. remove unnecessary tuned results.

@silverbullet233

Copy link
Copy Markdown
Contributor Author

@alexey-milovidov please take a look if you are available, thx

@nickitat nickitat self-assigned this Feb 13, 2023
@nickitat

nickitat commented Feb 13, 2023 •

Copy link
Copy Markdown
Member

I constantly get error on Q32:

30092059992	StarRocks-3.0.0-preview/storage/
30092059992	total
+----------+
| count(*) |
+----------+
| 99997497 |
+----------+
query0,0.01,0.01,0.01
query1,0.06,0.02,0.02
query2,0.08,0.05,0.05
query3,1.27,0.08,0.07
query4,0.29,0.28,0.30
query5,1.02,0.69,0.69
query6,0.02,0.01,0.02
query7,0.02,0.02,0.02
query8,0.45,0.41,0.41
query9,1.43,1.38,1.36
query10,0.10,0.08,0.08
query11,0.11,0.09,0.09
query12,0.52,0.50,0.50
query13,0.88,0.88,0.88
query14,0.78,0.70,0.74
query15,0.35,0.36,0.33
query16,1.25,1.24,1.28
query17,0.41,0.40,0.39
query18,2.42,2.21,2.30
query19,0.01,0.00,0.01
query20,11.03,1.10,1.10
query21,10.82,1.83,4.50
query22,24.78,18.91,15.67
query23,18.26,1.77,1.37
query24,0.40,0.08,0.05
query25,0.88,0.13,0.12
query26,0.04,0.06,0.04
query27,12.67,1.38,5.26
query28,10.50,10.31,10.13
query29,0.15,0.14,0.10
query30,3.09,0.37,0.38
query31,3.14,0.56,0.53
query32,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true]
,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true]
,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true]

query33,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true]
,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true]
,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true]

query34,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true]
,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true]
,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true]

query35,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true]
,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true]
,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true]

query36,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true]
,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true]
,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true]

query37,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true]
,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true]
,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true]

query38,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true]
,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true]
,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true]

query39,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true]
,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true]
,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true]

query40,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true]
,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true]
,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true]

query41,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true]
,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true]
,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true]

query42,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true]
,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true]
,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true]

[0.01,0.01,0.01],
[0.06,0.02,0.02],
[0.08,0.05,0.05],
[1.27,0.08,0.07],
[0.29,0.28,0.30],
[1.02,0.69,0.69],
[0.02,0.01,0.02],
[0.02,0.02,0.02],
[0.45,0.41,0.41],
[1.43,1.38,1.36],
[0.10,0.08,0.08],
[0.11,0.09,0.09],
[0.52,0.50,0.50],
[0.88,0.88,0.88],
[0.78,0.70,0.74],
[0.35,0.36,0.33],
[1.25,1.24,1.28],
[0.41,0.40,0.39],
[2.42,2.21,2.30],
[0.01,0.00,0.01],
[11.03,1.10,1.10],
[10.82,1.83,4.50],
[24.78,18.91,15.67],
[18.26,1.77,1.37],
[0.40,0.08,0.05],
[0.88,0.13,0.12],
[0.04,0.06,0.04],
[12.67,1.38,5.26],
[10.50,10.31,10.13],
[0.15,0.14,0.10],
[3.09,0.37,0.38],
[3.14,0.56,0.53],
[ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true] ],
,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true] ],
,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true] ],
],
[ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true] ],
,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true] ],
,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true] ],
],
[ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true] ],
,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true] ],
,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true] ],
],
[ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true] ],
,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true] ],
,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true] ],
],
[ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true] ],
,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true] ],
,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true] ],
],
[ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true] ],
,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true] ],
,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true] ],
],
[ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true] ],
,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true] ],
,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true] ],
],
[ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true] ],
,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true] ],
,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true] ],
],
[ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true] ],
,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true] ],
,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true] ],
],
[ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true] ],
,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true] ],
,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true] ],
],
[ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true] ],
,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true] ],
,ERROR 1064 (HY000) at line 1: Backend node not found. Check if any backend node is down.backend: [172.31.30.41 alive: true inBlacklist: true] ],
],

any ideas why?

be.INFO.log.20230213-154110.txt
be.out.txt
be.WARNING.log.20230213-155039.txt

@silverbullet233

Copy link
Copy Markdown
Contributor Author

Hi, @nickitat , very thanks for your feedback.

I see that your dictionary name contains ubuntu, it looks like you are testing on an ubuntu system. Currently, StarRocks only supports running on centos. The release package is also compiled on centos, and some strange problems may occur if it runs on other system. We have comments at the beginning of benchmark.sh about this.

FYI, here is the os version info in my test environment. I have tested many times in this environment and encountered no errors.

[ec2-user@ip-172-31-61-73 be]$ cat /etc/os-release
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"

so,could you verify it on Amazon Linux?

In addition, I notice that there are some warning messages in your be log.

W0213 17:10:30.636015 1705084 stream_load_executor.cpp:169] begin transaction failed, errmsg=Label [hits_tsv] has already been used.id=3c4ca296ab0a4d35-86aa78bee3aef9ae, job_id=-1, txn_id: -1, label=hits_tsv, db=hits

In order to avoid repeated load, StarRocks has a constraint that the label must be unique for each load, otherwise subsequent load job with the same label will be ignored. I just made an update and added the start time in the label to make it unique, which is convenient for you to perform multiple verifications on the same cluster.

please take a look again, thanks:)

@nickitat

Copy link
Copy Markdown
Member

We have comments at the beginning of benchmark.sh about this.

sorry, my bad

@nickitat
nickitat merged commit 48c1b52 into ClickHouse:main Feb 14, 2023
JelteF added a commit to JelteF/ClickBench that referenced this pull request Nov 15, 2024
Q28 contains backslashes, specifically the backslash in `\1` is very
important for the meaning of the query. However, both backslashes in
this query are accidentally removed in almost all benchmark scripts
because they used `read` instead of `read -r`.

This adds the `-r` flag to all the scripts that didn't have it yet, to
make sure that `read` interprets it as a raw `\` character instead of an
escape sequence.

With this change the REGEXP_REPLACE call doesn't always return the
literal string "1", but actually (as intended by the query) the referrer
domain. This change generally makes the query significantly slower to
execute, because now there are many more groups. I'm not sure how this
should be listed in the results UI. It might make sense to ignore the
query or something.

This problem was found independently in ClickHouse#47 and ClickHouse#72, but fixed in a
different way. I'm not sure if the fix that was made for starrocks in ClickHouse#72
is actually correct though, because it seems to fix the issue twice: It
replaces `\1` with `\\1`, but it also added the `-r` flag to `read`.

Apart from those issues there were already a few benchmarks that were
using the `-r` flag:
```
❯ rg 'while read -r'
elasticsearch/run.sh
5:cat 'queries.sql' | while read -r QUERY; do

starrocks/run.sh
8:cat queries.sql | while read -r query; do

doris/run.sh
8:while read -r query; do

crunchy-bridge-for-analytics/run.sh
22:    while read -r query; do
```
JelteF added a commit to JelteF/ClickBench that referenced this pull request Nov 15, 2024
Q28 contains backslashes, specifically the backslash in `\1` is very
important for the meaning of the query. However, both backslashes in
this query are accidentally removed in almost all benchmark scripts
because they used `read` instead of `read -r`.

This adds the `-r` flag to all the scripts that didn't have it yet, to
make sure that `read` interprets it as a raw `\` character instead of an
escape sequence.

With this change the REGEXP_REPLACE call doesn't always return the
literal string "1", but actually (as intended by the query) the referrer
domain. This change generally makes the query significantly slower to
execute, because now there are many more groups. I'm not sure how this
should be listed in the results UI. It might make sense to ignore the
query or something.

This problem was found independently in ClickHouse#47 and ClickHouse#72, but fixed in a
different way. I'm not sure if the fix that was made for starrocks in ClickHouse#72
is actually correct though, because it seems to fix the issue twice: It
replaces `\1` with `\\1`, but it also added the `-r` flag to `read`.

Apart from those issues there were already a few benchmarks that were
using the `-r` flag:
```
❯ rg 'while read -r'
elasticsearch/run.sh
5:cat 'queries.sql' | while read -r QUERY; do

starrocks/run.sh
8:cat queries.sql | while read -r query; do

doris/run.sh
8:while read -r query; do

crunchy-bridge-for-analytics/run.sh
22:    while read -r query; do
```
JelteF added a commit to JelteF/ClickBench that referenced this pull request Nov 15, 2024
Q28 contains backslashes, specifically the backslash in `\1` is very
important for the meaning of the query. However, both backslashes in
this query are accidentally removed in almost all benchmark scripts
because they used `read` instead of `read -r`.

This adds the `-r` flag to all the scripts that didn't have it yet, to
make sure that `read` interprets it as a raw `\` character instead of an
escape sequence.

With this change the REGEXP_REPLACE call doesn't always return the
literal string "1", but actually (as intended by the query) the referrer
domain. This change generally makes the query significantly slower to
execute, because now there are many more groups. I'm not sure how this
should be listed in the results UI. It might make sense to ignore the
query or something.

This problem was found independently in ClickHouse#47 and ClickHouse#72, but fixed in a
different way. I'm not sure if the fix that was made for starrocks in ClickHouse#72
is actually correct though, because it seems to fix the issue twice: It
replaces `\1` with `\\1`, but it also added the `-r` flag to `read`.

Apart from those issues there were already a few benchmarks that were
using the `-r` flag:
```
❯ rg 'while read -r'
elasticsearch/run.sh
5:cat 'queries.sql' | while read -r QUERY; do

starrocks/run.sh
8:cat queries.sql | while read -r query; do

doris/run.sh
8:while read -r query; do

crunchy-bridge-for-analytics/run.sh
22:    while read -r query; do
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants