Manticore crashes with `signal 11` when inserting data
Describe the bug Manticore crashes when inserting a large amount of data into index.
Manticore is running in rt mode with following tables:
CREATE TABLE redacted_aaa (
id bigint,
entity_id text stored,
name text indexed,
description text indexed,
notes text indexed,
number text indexed,
holder text indexed,
is_deleted bool,
schema string attribute,
source string attribute,
attributes json
) min_infix_len='2' index_exact_words='1' charset_table='non_cjk, U+47->0' min_word_len='2' blend_chars='@, /, +, -, ., _' blend_mode='trim_none, trim_both, skip_pure' morphology='stem_ru' min_stemming_len='3' expand_keywords='1'Data is inserted via (rather large?) batches of 500 records per single insert, and whole dataset contains about 100m rows splitted into 1-3 indexes. Crash happens randomly, data can be inserted without problems at all, or can crash at ~1-2% at random line.
Since it is prod instance, I'm afraid I can not give you our datasets, or test multiple (older?) manticore versions.
To Reproduce Steps to reproduce the behavior:
- Create the tables;
- Insert large dataset via batches;
- Randomly get a crash;
Expected behavior It should not crash.
Describe the environment:
searchd -v: Manticore 6.2.12 dc5144d35@230822- Running in official manticore docker image
manticoresearch/manticore:6.2.12
Messages from log files:
docker logs shows following:
rt: table redacted_aaa: diskchunk 7(8), segments 30 saved in 20.478723 (20.479030) sec, RAM saved/new 127530839/2099120 ratio 0.950000 (soft limit 127506841, conf limit 134217728)
rt: table redacted_bbb: diskchunk 5(6), segments 31 saved in 5.438967 (5.440291) sec, RAM saved/new 127699405/232936 ratio 0.950000 (soft limit 127506841, conf limit 134217728)
# ~20-30 of similar lines
Crash!!! Handling signal 11After that it restarts with:
binlog: replaying log /var/lib/manticore/binlog/binlog.011
binlog: table redacted_aaa: recovered from tid 23457 to tid 23487
binlog: table redacted_bbb: recovered from tid 5380 to tid 5418
binlog: replay stats: 316 commits; 0 updates, 0 reconfigure; 0 pq-add; 0 pq-delete; 0 pq-add-delete, 2 tables
binlog: finished replaying /var/lib/manticore/binlog/binlog.011; 85.2 MB in 0.551 sec
binlog: finished replaying total 1 in 0.552 secAdditional context While writing this issue, I came up with two ideas:
decrease batch size to maybe 50 rows per single insert;didn't help- try to turn off binlog (we do not need it anyway I think);
I'll try both options, but since crash is happening randomly - I couldnt guarantee it will work or not.
Any advices is greatly appreciated,
indextool --check on both indexes returns:
# other chunks gives same output
checking disk chunk, extension 16, 16(17)...
WARNING: secondary library not loaded; secondary index(es) disabled
checking schema...
checking dictionary...
checking data...
checking rows...
checking attribute blocks index...
checking kill-list...
checking docstore...
checking dead row map...
checking doc-id lookup...
check passed, 196.3 sec elapsed
check passed, 196.3 sec elapsedSource: manticoresoftware/manticoresearch