lboka/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/mrs/component-operation-guide-lts/mrs_01_1472.html
Yang, Tong 3f5759eed2 MRS comp-lts 2.0.38.SP20 version 
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com>
Co-authored-by: Yang, Tong <yangtong2@huawei.com>
Co-committed-by: Yang, Tong <yangtong2@huawei.com>
2023-01-19 17:08:45 +00:00

2.6 KiB
Raw Permalink Blame History

How Do I Configure Unsafe Memory in CarbonData?

Question

How do I configure unsafe memory in CarbonData?

Answer

In the Spark configuration, the value of spark.yarn.executor.memoryOverhead must be greater than the sum of (sort.inmemory.size.inmb + Netty offheapmemory required), or the sum of (carbon.unsafe.working.memory.in.mb + carbon.sort.inememory.storage.size.in.mb + Netty offheapmemory required). Otherwise, if off-heap access exceeds the configured executor memory, Yarn may stop the executor.

If spark.shuffle.io.preferDirectBufs is set to true, the netty transfer service in Spark takes off some heap memory (around 384 MB or 0.1 x executor memory) from spark.yarn.executor.memoryOverhead.

For details, see Configuring Executor Off-Heap Memory.

Parent topic: CarbonData FAQ