While we strive to give you perfect uptime, like other complex applications RethinkDB is not immune to crashing. Here are some tips on how to recover from a crash, how to submit a bug report, and how to maximize availability.
You may be able to check if the kernel’s out-of-memory killer is responsible for the crash by checking the system message buffer:
sudo dmesg | grep oom
This may show you messages similar to this:
rethinkdb invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
[<ffffffff8111d272>] ? oom_kill_process+0x82/0x2a0
If this is the case, you may be able to avoid crashes by changing RethinkDB’s cache size. For information on in-memory caches, how to check their current size, and how to change them, read Understanding RethinkDB memory requirements.
The log file’s location is dependent on your system configuration and how you started RethinkDB.
If you started rethinkdb on a terminal rather than from a startup script, it will log to the rethinkdb_data directory. By default it will write to log_file but this may be overridden with the --log-file startup option.
If your Linux system uses systemd, use journalctl to view the log:
journalctl -u rethinkdb@<instance>
If you installed RethinkDB through a package manager on a system that does not use systemd, then you may have to check where it’s configured to log. It’s very likely this will be in the /var/log/ directory (i.e., /var/log/rethinkdb).
The log may give you information as to what caused the crash.
If it doesn’t appear to be a memory issue and the log provides no clue, you can try asking for support on our official IRC channel, #rethinkdb on freenode or our Google Group. If your problem is a crash that we’ve seen before—or our users have—this may get you a quick answer.
We use Github for issue tracking: https://github.com/rethinkdb/rethinkdb/issues. If you want to report a suspected bug in RethinkDB, open an issue there.
The most important things for you to provide for us are:
The full output from rethinkdb --version, something like:
rethinkdb 1.13.3 (CLANG 5.1 (clang-503.0.40))
The full output from uname -a, something like:
Darwin rethink.local 13.3.0 Darwin Kernel Version 13.3.0:
Tue Jun 3 21:27:35 PDT 2014; root:xnu-2422.110.17~1/RELEASE_X86_64 x86_64
The backtrace from the crash, if it’s available in the logs.
Other things that might be helpful to us, if you have them:
rethinkdb._debug_table_status table (a “hidden” table in the rethinkdb system database)rethinkdb on startupIn the Data Explorer, the following command will output the contents of all the configuration/status tables and the most recent 50 lines of the logs table:
r.expr(["current_issues", "jobs", "stats", "server_config", "server_status",
"table_config", "table_status", "db_config", "cluster_config"]).map(
[r.row, r.db('rethinkdb').table(r.row).coerceTo('array')]
).coerceTo('object').merge(
{logs: r.db('rethinkdb').table('logs').limit(50).coerceTo('array')}
)
RethinkDB supports replication of data: every table in a database can be replicated as many times as you have servers in a cluster. Setting up replication is a simple operation with the web interface or the command line tool. For details, read Sharding and replication.
RethinkDB does not have fully automatic failover (yet), but if a server in a cluster crashes it can be manually removed from the cluster. In most cases, RethinkDB will recover from such a situation automatically. For information on this process, read Failover.