Elasticsearch is an open-source search engine. This section provides some suggestions on using Elasticsearch to help you better use CSS.
A single thread that sends bulk requests is unlikely to max out the indexing capability of a cluster. However, to maximize utilization of cluster resources and improve data processing efficiency, send data through multiple threads or processes.
By testing, you can determine the optimal number of threads for bulk requests of the same size. You can progressively increase the number of threads until the CPU is saturated in the cluster. You are advised to use the nodes stats API to view the CPU status of a node. View the os.cpu.percent, os.cpu.load_average.1m, os.cpu.load_average.5m, and os.cpu.load_average.15m parameter settings for more details. For information on how to use the nodes stats API and parameter descriptions, see https://www.elastic.co/guide/en/elasticsearch/reference/6.2/cluster-nodes-stats.html#os-stats.
For example, check whether the CPU is saturated if two threads are used during execution of bulk requests. If it is not saturated, increase the number of threads. If the load or CPU is saturated when the number of threads reaches N, you are advised to use N threads (the optimal number according to your testing) to execute bulk requests and improve indexing efficiency.
By default, each shard is automatically refreshed once per second. However, the refresh frequency is not applicable to all scenarios. If you use Elasticsearch to index a large number of log files and want to increase the indexing speed instead of obtaining near-real-time search performance, you can reduce the refresh frequency of each index.
PUT /my_logs
{
"settings": {
"refresh_interval": "30s"
}
}
If you need to import a large amount of data at a time, it is recommended that you disable refresh and replicas by setting refresh_interval to -1 and number_of_replicas to 0. After you import the data, set refresh_interval and number_of_replicas to the original values.
When you create index data, you are advised to specify the number of shards and replicas. Otherwise, default settings (five shards and one replica) will be used.
The shard quantity is strongly relevant to the indexing speed. Too many or too few shards will lead to slow indexing. If you specify too many shards, numerous files will be opened during retrieval, slowing down the communication between servers. If you specify too few shards, the index size of a single shard may be too large, slowing down the indexing speed.
Specify the shard quantity based on the node quantity, disk quantity, and index size. It is recommended that the size of a single shard not exceed 30 GB. The shard size is calculated using the following formula: Size of a shard = Total amount of data/Shard quantity
PUT /my_index
{
"settings": {
"number_of_shards": 1,
"number_of_replicas": 0
}
}
Elasticsearch relies on Lucene to index and store data and it suits dense data, which means that all documents have the same field.
Do not put documents that have different data structures into the same index. You can consider creating some smaller indices and use fewer shards to store the documents into the indices.
It may save time and efforts to put different types of documents into an index. However, be aware that Elasticsearch does not store documents based on type. Therefore, putting different types into one index will slow down indexing. If your document types do not have similar mappings, put them in the same index.
Avoid two different types that have fields of the same name but different mappings.
You are advised to create indices to store time-related data, such as log data, by time range, instead of storing all data in a super large index.
For example, you can store data in an index named by year (example: logs_2014) or by month (example: logs_2014-10). When the volume of data becomes very large, you can store data in an index named by day (example: logs_2014-10-24).
Creating indices by time range has the following advantages:
You can flexibly set the number of shards and replicas for each index created by time range so that there is no need to set a large shard at the beginning. After the cluster capacity is expanded, the time range can be set to adapt to the cluster scale.
DELETE /logs_2014-09
Combine multiple segments in the logs_2014-09-30 index into a shard, improving the query efficiency.
PUT /logs_2014-09-30/_settings
{ "number_of_replicas": 0 }
POST /logs_2014-09-30/_forcemerge?max_num_segments=1
PUT /logs_2014-09-30/_settings
{ "number_of_replicas": 1 }
Versions later than 7.x
PUT /logs_2014-09-30/_settings
{ "number_of_replicas": 0 }
POST /logs_2014-09-30/_forcemerge
{
"max_num_segments":1
}
PUT /logs_2014-09-30/_settings
{ "number_of_replicas": 1 }
In Elasticsearch, the string field is divided into two new data types: text used for full-text search and keyword used for keyword search.
You are advised to configure exact-value fields without word segmentation, such as tags or enumerated values, as the keyword type.
Versions earlier than 7.x
PUT my_index1
{
"mappings": {
"my_type": {
"properties": {
"tags": {
"type": "keyword"
},
"full_name": {
"type": "text"
}
}
}
}
}
Versions later than 7.x
PUT my_index1
{
"mappings": {
"properties": {
"tags": {
"type": "keyword"
},
"full_name": {
"type": "text"
}
}
}
}
Aggregated statistics based on the text field is not a common requirement. In Elasticsearch, to use aggregated statistics based on the text field, you need to enable fielddata (disabled by default). However, enabling fielddata will consume significant memory.
You are advised to conduct multifield mapping on the sub-word string to a text field for full-text search and a keyword field for aggregated statistics.
Versions earlier than 7.x
PUT my_index2
{
"mappings": {
"my_type": {
"properties": {
"full_name": {
"type": "text",
"fields": {
"raw": {
"type": "keyword"
}
}
}
}
}
}
}
Versions later than 7.x
PUT my_index2
{
"mappings": {
"properties": {
"full_name": {
"type": "text",
"fields": {
"raw": {
"type": "keyword"
}
}
}
}
}
}
Elasticsearch allows you to use index templates to control settings and mappings of certain created indices, for example, controlling the shard quantity to 1 and disabling the _all field. You can use the index template to control the settings you want to apply to the created indices.
In the following example, the index matching logstash-* uses the my_logs template, and the priority value of the my_logs template is 1.
Versions earlier than 7.x
PUT /_template/my_logs
{
"template": "logstash-*",
"order": 1,
"settings": {
"number_of_shards": 1
},
"mappings": {
"_default_": {
"_all": {
"enabled": false
}
}
},
"aliases": {
"last_3_months": {}
}
}
Versions later than 7.x
PUT /_template/my_logsa
{
"index_patterns": ["logstasaah-*"],
"order": 1,
"settings": {
"number_of_shards": 1
},
"mappings": {
"properties": {
"_all": {
"enabled": false
}
}
},
"aliases": {
"last_3_months": {}
}
}
Elasticsearch replicas provide high availability during runtime, which ensures service continuity even when sporadic data loss occurs.
However, replicas do not protect against failures. In case of a failure, you need a backup of your cluster so that you can restore data.
To back up cluster data, create snapshots and save them in OBS buckets. This backup process is "smart". You are advised to use your first snapshot to store a copy of your data. All subsequent snapshots will save the differences between the existing snapshots and the new data. As the number of snapshots increases, backups are added or deleted accordingly. This means that subsequent backups will take a shorter time because only a small volume of data needs to be transferred.
Filters are important because they are fast. They do not calculate relevance (skipping the entire scoring phase) and are easily cached.
Usually, when you look for an exact value, you will not want to score the query. You would want to include/exclude documents, so you will use a constant_score query to execute the term query in a non-scoring mode and apply a uniform score of one.
GET /my_store/products/_search
{
"query" : {
"constant_score" : {
"filter" : {
"term" : {
"city" : "London"
}
}
}
}
}
In the scenario where a large amount of data is returned, the query-then-fetch process supports pagination with the from and size parameters, but within limits. Results are sorted on each shard before being returned. However, with larger from values, the sorting process can become very heavy, using vast amounts of CPU, memory, and bandwidth. For this reason, deep pagination is not recommended.
You can use a scroll query to retrieve large numbers of documents from Elasticsearch efficiently, without affecting system performance. Scrolling allows you to do an initial search and to keep pulling batches of results from Elasticsearch until there are no more results left.
In general, a filter will outperform a scoring query.
When used in filtering context, the query is said to be a non-scoring or filtering query. That is, the query simply asks the question: Does this document match? The answer is always a simple, binary yes|no.
Typical filtering cases are listed as follows:
When used in a querying context, the query becomes a "scoring" query. Similar to the non-scoring query, this query also determines if a document matches and how well the document matches. A typical use for a scoring query is to find documents:
Queries can become quite complex. Especially, when they are combined with different analyzers and field mappings, they can become a little difficult to follow. You can use the validate-query API to check whether a query is valid.
For example, on the Kibana Console page, run the following command to check whether the query is valid. In this example, the validate request tells you that the query is invalid.
Versions earlier than 7.x
GET /gb/tweet/_validate/query
{
"query": {
"tweet" : {
"match" : "really powerful"
}
}
}
Versions later than 7.x
GET /gb/tweet/_validate/query
{
"query": {
"productName" : {
"match" : "really powerful"
}
}
}
The response to the preceding validate request tells us that the query is invalid. To find out why it is invalid, add the explain parameter to the query string and execute the following command.
Versions earlier than 7.x
GET /gb/tweet/_validate/query?explain
{
"query": {
"tweet" : {
"match" : "really powerful"
}
}
}
Versions later than 7.x
GET /gb/tweet/_validate/query?explain
{
"query": {
"productName" : {
"match" : "really powerful"
}
}
}
According to the command output shown in the following, the type of query (match) is mixed up with the name of the field (tweet).
{
"valid": false,
"error": "org.elasticsearch.common.ParsingException: no [query] registered for [tweet]"
}
Using the explain parameter has the added advantage of returning a human-readable description of the (valid) query, which helps in understanding exactly how CSS interprets your query.