forked from docs/doc-exports
Yang, Tong
3f5759eed2
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
203 lines
16 KiB
HTML
203 lines
16 KiB
HTML
<a name="mrs_01_24287"></a><a name="mrs_01_24287"></a>
|
|
|
|
<h1 class="topictitle1">Adaptive MV Usage in ClickHouse</h1>
|
|
<div id="body32001227"><div class="section" id="mrs_01_24287__en-us_topic_0000001219029825_section10515183434410"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_24287__en-us_topic_0000001219029825_p1456163595016">Materialized views (MVs) are used in ClickHouse to save the precomputed result of time-consuming operations. When querying data, you can query the materialized views rather than the original tables, thereby quickly obtaining the query result.</p>
|
|
<p id="mrs_01_24287__en-us_topic_0000001219029825_p1198821535212">Currently, MVs are not easy to use in ClickHouse. Users can create one or more MVs based on the original table data as required. Once multiple MVs are created, you need to identify which MV is used and convert the query statement of an original table to that of an MV. In this way, the querying process is inefficient and prone to errors.</p>
|
|
<p id="mrs_01_24287__en-us_topic_0000001219029825_p13129132745611">The problem mentioned above is readily solved since the adoption of adaptive MVs. When querying an original table, the corresponding MV of this table will be queried, which greatly improves the usability and efficiency of ClickHouse.</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_24287__en-us_topic_0000001219029825_section20236132619318"><h4 class="sectiontitle">Matching Rules of Adaptive MVs</h4><p id="mrs_01_24287__en-us_topic_0000001219029825_p66415314416">To ensure that the SQL statement for querying an original table can be automatically converted to that for querying the corresponding MV, the following matching rules must be met:</p>
|
|
<ul id="mrs_01_24287__en-us_topic_0000001219029825_ul1595749132912"><li id="mrs_01_24287__en-us_topic_0000001219029825_li69584919294">The table to be queried using an SQL statement must be associated with an MV.</li><li id="mrs_01_24287__en-us_topic_0000001219029825_li191601631103319">The AggregatingMergeTree engine must be used with MVs.</li><li id="mrs_01_24287__en-us_topic_0000001219029825_li19885165013293">Both the SELECT clause of SQL and MVs must contain aggregate functions.</li><li id="mrs_01_24287__en-us_topic_0000001219029825_li4858135472910">If the SQL query contains a GROUP BY clause, MVs must also contain this clause.</li><li id="mrs_01_24287__en-us_topic_0000001219029825_li2536145710291">If an MV contains a WHERE clause of SQL, the WHERE clause must be the same as that of the MV. This also applies to the PREWHERE and HAVING clauses.</li><li id="mrs_01_24287__en-us_topic_0000001219029825_li141101605308">Fields to be queried using the SQL statements must exist in the MVs.</li><li id="mrs_01_24287__en-us_topic_0000001219029825_li1916920343019">If multiple MVs meet the preceding requirements, the SQL statement for querying the original table will be used.</li></ul>
|
|
<p id="mrs_01_24287__en-us_topic_0000001219029825_p9337162720117">For details about common matching failures of adaptive MVs, see <a href="#mrs_01_24287__en-us_topic_0000001219029825_section332911341804">Common Matching Failures of MVs</a>.</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_24287__en-us_topic_0000001219029825_section973532716111"><h4 class="sectiontitle">Using Adaptive MVs</h4><p id="mrs_01_24287__en-us_topic_0000001219029825_p189245439110">In the following operations, <strong id="mrs_01_24287__en-us_topic_0000001219029825_b5716183124315">local_table</strong> is the original table and <strong id="mrs_01_24287__en-us_topic_0000001219029825_b1517181112434">view_table</strong> is the MV created based on <strong id="mrs_01_24287__en-us_topic_0000001219029825_b1329318488437">local_table</strong>. Change the table creation and query statements based on the site requirements.</p>
|
|
<ol id="mrs_01_24287__en-us_topic_0000001219029825_ol43756191752"><li id="mrs_01_24287__en-us_topic_0000001219029825_li72621951086"><span>Use the ClickHouse client to connect to the default database. For details, see <a href="mrs_01_2345.html">Using ClickHouse from Scratch</a>.</span></li><li id="mrs_01_24287__en-us_topic_0000001219029825_li1037516198516"><span>Run the following table creation statements to create the original table <strong id="mrs_01_24287__en-us_topic_0000001219029825_b151021049175818">local_table</strong>.</span><p><pre class="screen" id="mrs_01_24287__en-us_topic_0000001219029825_screen133810201148">CREATE TABLE local_table
|
|
(
|
|
id String,
|
|
city String,
|
|
code String,
|
|
value UInt32,
|
|
create_time DateTime,
|
|
age UInt32
|
|
)
|
|
ENGINE = MergeTree
|
|
PARTITION BY toDate(create_time)
|
|
ORDER BY (id, city, create_time);</pre>
|
|
</p></li><li id="mrs_01_24287__en-us_topic_0000001219029825_li18338142010145"><span>Create the MV <strong id="mrs_01_24287__en-us_topic_0000001219029825_b5966112215916">view_table</strong> based on <strong id="mrs_01_24287__en-us_topic_0000001219029825_b16656203915592">local_table</strong>.</span><p><pre class="screen" id="mrs_01_24287__en-us_topic_0000001219029825_screen1021511381158">CREATE MATERIALIZED VIEW view_table
|
|
ENGINE = AggregatingMergeTree
|
|
PARTITION BY toDate(create_time)
|
|
ORDER BY (id, city, create_time)
|
|
AS SELECT
|
|
create_time,
|
|
id,
|
|
city,
|
|
uniqState(code),
|
|
sumState(value) AS value_new,
|
|
minState(create_time) AS first_time,
|
|
maxState(create_time) AS last_time
|
|
FROM local_table
|
|
WHERE create_time >= toDateTime('2021-01-01 00:00:00')
|
|
GROUP BY id, city, create_time;</pre>
|
|
</p></li><li id="mrs_01_24287__en-us_topic_0000001219029825_li1821553831513"><span>Insert data to the <strong id="mrs_01_24287__en-us_topic_0000001219029825_b1728114912016">local_table</strong> table.</span><p><pre class="screen" id="mrs_01_24287__en-us_topic_0000001219029825_screen1336123221614">INSERT INTO local_table values('1','zzz','code1',1,toDateTime('2021-01-02 00:00:00'), 10);
|
|
INSERT INTO local_table values('2','kkk','code2',2,toDateTime('2020-01-01 00:00:00'), 20);
|
|
INSERT INTO local_table values('3','ccc','code3',3,toDateTime('2022-01-01 00:00:00'), 30);</pre>
|
|
</p></li><li id="mrs_01_24287__en-us_topic_0000001219029825_li133611932171614"><span>Run the following command to enable the adaptive MVs.</span><p><pre class="screen" id="mrs_01_24287__en-us_topic_0000001219029825_screen6355152741714">set adaptive_materilized_view = 1;</pre>
|
|
<div class="note" id="mrs_01_24287__en-us_topic_0000001219029825_note533724261812"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_24287__en-us_topic_0000001219029825_p1333816425189">If the <strong id="mrs_01_24287__en-us_topic_0000001219029825_b162578378211">adaptive_materilized_view</strong> parameter is set to <strong id="mrs_01_24287__en-us_topic_0000001219029825_b17674194215219">1</strong>, the adaptive MVs are enabled. If it is set to <strong id="mrs_01_24287__en-us_topic_0000001219029825_b289013371142">0</strong>, the adaptive MVs are disabled. The default value is <strong id="mrs_01_24287__en-us_topic_0000001219029825_b87220117620">0</strong>. <strong id="mrs_01_24287__en-us_topic_0000001219029825_b1933115258183">set adaptive_materilized_view = 1;</strong> is a session-level command and needs to be reset each time the client connects to the server.</p>
|
|
</div></div>
|
|
</p></li><li id="mrs_01_24287__en-us_topic_0000001219029825_li1961113141710"><a name="mrs_01_24287__en-us_topic_0000001219029825_li1961113141710"></a><a name="en-us_topic_0000001219029825_li1961113141710"></a><span>Query data in the <strong id="mrs_01_24287__en-us_topic_0000001219029825_b945012012121">local_table</strong> table.</span><p><pre class="screen" id="mrs_01_24287__en-us_topic_0000001219029825_screen12332215214">SELECT sum(value)
|
|
FROM local_table
|
|
WHERE create_time >= toDateTime('2021-01-01 00:00:00')
|
|
┌─sumMerge(value_new)─┐
|
|
│ 4 │
|
|
└─────────────────────┘</pre>
|
|
</p></li><li id="mrs_01_24287__en-us_topic_0000001219029825_li6342296220"><span>Run the <strong id="mrs_01_24287__en-us_topic_0000001219029825_b18652162945213">explain syntax</strong> command to view the execution plan of the SQL statement in step <a href="#mrs_01_24287__en-us_topic_0000001219029825_li1961113141710">6</a>. According to the query result, <strong id="mrs_01_24287__en-us_topic_0000001219029825_b1737710307365">view_table</strong> is queried.</span><p><pre class="screen" id="mrs_01_24287__en-us_topic_0000001219029825_screen12662181118239">EXPLAIN SYNTAX
|
|
SELECT sum(value)
|
|
FROM local_table
|
|
WHERE create_time >= toDateTime('2021-01-01 00:00:00')
|
|
┌─explain────────────────────┐
|
|
│ SELECT sumMerge(value_new) │
|
|
│ FROM default.view_table │
|
|
└────────────────────────────┘</pre>
|
|
</p></li></ol>
|
|
</div>
|
|
<div class="section" id="mrs_01_24287__en-us_topic_0000001219029825_section332911341804"><a name="mrs_01_24287__en-us_topic_0000001219029825_section332911341804"></a><a name="en-us_topic_0000001219029825_section332911341804"></a><h4 class="sectiontitle">Common Matching Failures of MVs</h4><ul id="mrs_01_24287__en-us_topic_0000001219029825_ul194115421009"><li id="mrs_01_24287__en-us_topic_0000001219029825_li163995374281">When creating an MV, the aggregate functions must contain the State suffix. Otherwise, the corresponding MV cannot be matched. Example:<pre class="screen" id="mrs_01_24287__en-us_topic_0000001219029825_screen18943502415"># # The MV agg_view is created based on the original table test_table. However, the count aggregate function does not contain the State suffix.
|
|
CREATE MATERIALIZED VIEW agg_view
|
|
ENGINE = AggregatingMergeTree
|
|
PARTITION BY toDate(create_time)
|
|
ORDER BY (id)
|
|
AS SELECT
|
|
create_time,
|
|
id,
|
|
<strong id="mrs_01_24287__en-us_topic_0000001219029825_b116321861618">count</strong>(id)
|
|
FROM test_table
|
|
GROUP BY id,create_time;
|
|
|
|
# To ensure that the MV can be matched, the count aggregate function for creating the MV must contain the State suffix. The correct example is as follows:
|
|
CREATE MATERIALIZED VIEW agg_view
|
|
ENGINE = AggregatingMergeTree
|
|
PARTITION BY toDate(create_time)
|
|
ORDER BY (id)
|
|
AS SELECT
|
|
create_time,
|
|
id,
|
|
<strong id="mrs_01_24287__en-us_topic_0000001219029825_b168971722181610">countState</strong>(id)
|
|
FROM test_table
|
|
GROUP BY id,create_time;</pre>
|
|
</li><li id="mrs_01_24287__en-us_topic_0000001219029825_li3589432151210">Only if the WHERE clause of the statement for querying an original table is completely the same as that in an MV can the MV be matched.<p id="mrs_01_24287__en-us_topic_0000001219029825_p1413123417121"><a name="mrs_01_24287__en-us_topic_0000001219029825_li3589432151210"></a><a name="en-us_topic_0000001219029825_li3589432151210"></a>For example, if the WHERE clause of the original table statement is <strong id="mrs_01_24287__en-us_topic_0000001219029825_b13665918191114">where a=b</strong> while the WHERE clause of the MV is <strong id="mrs_01_24287__en-us_topic_0000001219029825_b121562791117">where b=a</strong>, the corresponding MV cannot be matched.</p>
|
|
<div class="p" id="mrs_01_24287__en-us_topic_0000001219029825_p51210833811">However, if the statement for querying the original table does not contain the database name, the corresponding MV can be matched. Example:<pre class="screen" id="mrs_01_24287__en-us_topic_0000001219029825_screen10546212184618"># The MV view_test is created based on db_test.table_test. The WHERE clause for querying the original table contains the database name db_test.
|
|
CREATE MATERIALIZED VIEW db_test.view_test ENGINE = AggregatingMergeTree ORDER BY phone AS
|
|
SELECT
|
|
name,
|
|
phone,
|
|
uniqExactState(class) as uniq_class,
|
|
sumState(CRC32(phone))
|
|
FROM db_test.table_test
|
|
WHERE (class, name) GLOBAL IN
|
|
(
|
|
SELECT class, name FROM <strong id="mrs_01_24287__en-us_topic_0000001219029825_b5650923111915">db_test.</strong>table_test
|
|
WHERE
|
|
name = 'zzzz'
|
|
AND class = 'calss one'
|
|
)
|
|
GROUP BY
|
|
name, phone;
|
|
# If the WHERE clause does not contain the database name db_test, the corresponding MV will be matched.
|
|
USE db_test;
|
|
EXPLAIN SYNTAX
|
|
SELECT
|
|
name,
|
|
phone,
|
|
uniqExact(class) as uniq_class,
|
|
sum(CRC32(phone))
|
|
FROM table_test
|
|
WHERE (class, name) GLOBAL IN
|
|
(
|
|
SELECT class, name FROM table_test
|
|
WHERE
|
|
name = 'zzzz'
|
|
AND class = 'calss one'
|
|
)
|
|
GROUP BY
|
|
name, phone;</pre>
|
|
</div>
|
|
</li><li id="mrs_01_24287__en-us_topic_0000001219029825_li156554352380">If the GROUP BY clause contains functions, the corresponding MV can be matched only when the column field names in the functions are the same as those in an original table. Example:<pre class="screen" id="mrs_01_24287__en-us_topic_0000001219029825_screen32625233535"># Create the MV agg_view based on test_table.
|
|
CREATE MATERIALIZED VIEW agg_view
|
|
ENGINE = AggregatingMergeTree
|
|
PARTITION BY toDate(create_time)
|
|
ORDER BY (id, city, create_time)
|
|
AS SELECT
|
|
create_time,
|
|
id,
|
|
city,
|
|
value as value1,
|
|
uniqState(code),
|
|
sumState(value) AS value_new,
|
|
minState(create_time) AS first_time,
|
|
maxState(create_time) AS last_time
|
|
FROM test_table
|
|
GROUP BY <strong id="mrs_01_24287__en-us_topic_0000001219029825_b183979161569">id, city, create_time, value1 % 2, value1</strong>;
|
|
# The corresponding MV can be matched if the statement is as follows:
|
|
SELECT uniq(code) FROM test_table GROUP BY <strong id="mrs_01_24287__en-us_topic_0000001219029825_b17586133211565">id, city, value1 % 2</strong>;
|
|
# The corresponding MV cannot be matched if the statement is as follows:
|
|
SELECT uniq(code) FROM test_table GROUP BY <strong id="mrs_01_24287__en-us_topic_0000001219029825_b686518905718">id, city, value % 2;</strong> </pre>
|
|
</li><li id="mrs_01_24287__en-us_topic_0000001219029825_li338564453818">In a created MV, the FROM clause cannot be a SELECT statement. Otherwise, the corresponding MV will fail to be matched. In the following example, the FROM clause is a SELECT statement. In this case, the corresponding MV cannot be matched.<pre class="screen" id="mrs_01_24287__en-us_topic_0000001219029825_screen9294204913573">CREATE MATERIALIZED VIEW agg_view
|
|
ENGINE = AggregatingMergeTree
|
|
PARTITION BY toDate(create_time)
|
|
ORDER BY (id)
|
|
AS SELECT
|
|
create_time,
|
|
id,
|
|
countState(id)
|
|
FROM
|
|
<strong id="mrs_01_24287__en-us_topic_0000001219029825_b941110368288">(SELECT id, create_time FROM test_table)</strong>
|
|
GROUP BY id,create_time;</pre>
|
|
</li><li id="mrs_01_24287__en-us_topic_0000001219029825_li19382175193812">When querying original tables or creating MVs, an aggregate function cannot be used together with another aggregate function or a common function. Example:<pre class="screen" id="mrs_01_24287__en-us_topic_0000001219029825_screen155041915924"># Case 1: Multiple aggregate functions are used when querying an original table.
|
|
# Create an MV.
|
|
CREATE MATERIALIZED VIEW agg_view
|
|
ENGINE = AggregatingMergeTree
|
|
PARTITION BY toDate(create_time)
|
|
ORDER BY (id)
|
|
AS SELECT
|
|
create_time,
|
|
id,
|
|
countState(id)
|
|
FROM test_table
|
|
GROUP BY id,create_time;
|
|
# Two aggregate functions are used when querying the original table, leading to the MV matching failure.
|
|
SELECT <strong id="mrs_01_24287__en-us_topic_0000001219029825_b928155182912">count(id) + count(id)</strong> FROM test_table;
|
|
# Case 2: Multiple aggregate functions are used when creating an MV.
|
|
# Two countState(id) functions are used when creating the MV, leading to the MV matching failure.
|
|
CREATE MATERIALIZED VIEW agg_view
|
|
ENGINE = AggregatingMergeTree
|
|
PARTITION BY toDate(create_time)
|
|
ORDER BY (id)
|
|
AS SELECT
|
|
create_time,
|
|
id,
|
|
(<strong id="mrs_01_24287__en-us_topic_0000001219029825_b1011642512181">countState(id) + countState(id)</strong>) AS new_count
|
|
FROM test_table
|
|
GROUP BY id,create_time;
|
|
# The corresponding MV cannot be matched when querying the original table.
|
|
SELECT new_count FROM test_table;</pre>
|
|
<div class="p" id="mrs_01_24287__en-us_topic_0000001219029825_p165055231826">However, if the parameter of an aggregate function is the combination operation of fields, the corresponding MV can be matched.<pre class="screen" id="mrs_01_24287__en-us_topic_0000001219029825_screen0460173016211">CREATE MATERIALIZED VIEW agg_view
|
|
ENGINE = AggregatingMergeTree
|
|
PARTITION BY toDate(create_time)
|
|
ORDER BY (id)
|
|
AS SELECT
|
|
create_time,
|
|
id,
|
|
countState(id + id)
|
|
FROM test_table
|
|
GROUP BY id,create_time;
|
|
# The corresponding MV can be matched when querying the original table.
|
|
SELECT count(<strong id="mrs_01_24287__en-us_topic_0000001219029825_b158815494466">id + id</strong>) FROM test_table;</pre>
|
|
</div>
|
|
</li></ul>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_2344.html">Using ClickHouse</a></div>
|
|
</div>
|
|
</div>
|
|
|