vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/mrs/component-operation-guide/mrs_01_1472.html
Yang, Tong 6182f91ba8 MRS component operation guide_normal 2.0.38.SP20 version 
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com>
Co-authored-by: Yang, Tong <yangtong2@huawei.com>
Co-committed-by: Yang, Tong <yangtong2@huawei.com>
2022-12-09 14:55:21 +00:00

2.1 KiB
Raw Permalink Blame History

How Do I Configure Unsafe Memory in CarbonData?

Question

How do I configure unsafe memory in CarbonData?

Answer

In the Spark configuration, the value of spark.yarn.executor.memoryOverhead must be greater than the sum of (sort.inmemory.size.inmb + Netty offheapmemory required), or the sum of (carbon.unsafe.working.memory.in.mb + carbon.sort.inememory.storage.size.in.mb + Netty offheapmemory required). Otherwise, if off-heap access exceeds the configured executor memory, Yarn may stop the executor.

If spark.shuffle.io.preferDirectBufs is set to true, the netty transfer service in Spark takes off some heap memory (around 384 MB or 0.1 x executor memory) from spark.yarn.executor.memoryOverhead.

For details, see Configuring Executor Off-Heap Memory.

Parent topic: CarbonData FAQ