Scenario:

All the three nodes of Galera cluster uses 98 percent of memory and swapping heavily, no DML could be performed.

Investigation:

From the error log on the node 3 the below error has been notified:

Detected table cache mutex contention at instance 1: 60% waits. Additional table cache instance activated. Number of instances after activation: 2.

Querying the global status for the variable table_open_cache_misses and table_open_cache_overflows shows a high cache misses and overflows.

The table_open_cache variable current value is 600 during the bottleneck.

Solution:

The type of operation performed form the application will employ many concurrent sessions and batch like insert which will require a high table cache, hence increasing the open_table_cache to a value of 2000 solved the issue.