Alex Stetsenko

MySQL’s show full processlist is a great way to monitor and diagnose what is going on with the database server at certain moment.

However often it is needed to have an aggregated view or evaluate only specific type of users and\or queries.

The solution turned out to be pretty simple since show full processlist is just an alias for a regular query to processlist table within information_schema database.

So the following queries are equivalent:

SHOW FULL PROCESSLIST;
 
SELECT * FROM information_schema.processlist;

As result it is possible to present the data from process list any convenient way. There are some basic examples below.

-- display number of connections for each user
SELECT `USER`, COUNT(*) FROM information_schema.processlist
GROUP BY `USER`;
 
-- display number of connections for each host
SELECT `HOST`, COUNT(*) FROM information_schema.processlist
GROUP BY `HOST`;
 
-- display root user activity
SELECT * FROM information_schema.processlist
WHERE `USER` = 'root';
 
-- display processes associated with SELECT queries
SELECT * FROM information_schema.processlist
WHERE `INFO` LIKE 'SELECT %';
 
-- display average query time for each database
SELECT `DB`, AVG(`TIME`) FROM information_schema.processlist
GROUP BY `DB`;

Often it is quite reasonable to have the database to take care of some audit related columns like date added or updated.
Unfortunately MySQL does NOT allow functions within table definitions like let’s say MS SQL does, e.g.:

CREATE TABLE t (f datetime NOT NULL DEFAULT now());

The only work around is to use timestamp column instead of date or datetime ones. There can be only one auto set timestamp column though.
So I normally use it to store date updated and save the date created manually.

CREATE TABLE t
(
  name VARCHAR(20),
  date_updated TIMESTAMP NOT NULL
               DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
)

Also there are two things to keep in mind:

• date_updated will be set based on MySQL server time. This is especially important if other data is stored using GMT or non-server time zone.
• date_updated must NOT be a part of update or insert query.

-- set date_updated to current time
INSERT INTO t (name) VALUES ('test');
UPDATE t SET name = 'test';
 
-- override date_updated to certain value
INSERT INTO t (name, date_updated)
       VALUES ('test', '2011-08-02 20:00:00');
UPDATE t SET name = 'test',
             date_updated = '2011-08-02 20:00:00';

Alternatively it is possible to use triggers to have as many auto updated datetime columns as needed. However personally I prefer to avoid triggers as much as possible. Mostly because often triggers are misused and as result application complexity is increased for no reason.

Apparently there is a pretty straight forward way to recover MySQL password. Although server root access is mandatory it is great solution in certain cases like recovering password of abandoned test server or when server was configured by someone else etc.

The key for success is to be able to (re)start MySQL using -skip-grant-tables option. In that case MySQL won’t ask for password.
Then simply reset MySQL password and restart MySQL ordinarily.

Details for each step are described by Vivek Gite in Recover MySQL root Password.

There could be some cases when the general query log has to be kept running. This includes development stage, different flavors of debugging and query analysis and so on. However certain queries do not need to be stored there even though the logging is on. One of the examples is setting user password – it will be logged in clear text.

A possible workaround is to turn off the general logging temporarily. MySQL 5.1 and higher allows to do it without the service restart.

CREATE USER john;
SET SESSION sql_log_off = 1;
SET PASSWORD FOR john = PASSWORD('smith');
SET SESSION sql_log_off = 0;

However SUPER privilege is required to set the session variable sql_log_off.

Apparently some web applications do store uploaded files in the database as blob field by default (for instance it is true for Mantis bug tracker). While it is acceptable (but not too common?) to store files in the database, it may cause certain problems like uploading large files (particularly 1Mb+).

Normally the first thing to look at would be the actual application configuration. OK, Mantis is configured to accept files up to 5Mb by default. I.e. no problems here. Wait… but PHP is configured to accept files up to 2Mb by default… So where does 1Mb restriction come from?

The answer is that it comes from the database. MySQL has limits on packet size. The largest possible packet that can be transmitted to or from a MySQL 5.1 server or client is 1Gb. However by default it is 1Mb! As result any packets larger than 1Mb will cause packet too large error.

There is max_allowed_packet option in the configuration which allows to set the packet size. For example:

[mysqld]
max_allowed_packet=16M

Recently I’ve run into another MySQL bug. This time it is related to temporary tables. The bug was reported as #12257.

Basically select * from tmpTable does not work within stored procedure if any column is altered since the table has been originally created. So MySQL keeps converting * to a old column list while some columns may not exist anymore.

A simple workaround allows to eliminate impact of this bug as well as using select * is not recommended anyway. However the most surprising part of this story that this bug cannot be resolved for more than 5 years: it was reported in 2005.

It is still common to store IP addresses as a varchar(15) field though it is possible to use integer type instead. Unlike the varchar type, integer has fixed size and uses only 4 bytes.

INET_ATON() is used to convert an IP address to a number and INET_NTOA() – for the reverse operation.

SELECT INET_ATON('127.0.0.1');
SELECT INET_NTOA(2130706433);

It is important to use INT UNSIGNED with INET_ATON() so that IP addresses for which the first octet is greater than 127 is stored correctly.

Also PHP has similar functions – ip2long() and long2ip(). However ip2long() function may return negative results in certain cases. To make it always positive unsigned intereger ip2long() call has to be paired with printf() or sprintf() function:

printf('%u', ip2long('128.0.0.1');

Thus plan the IP address column datatype accordingly.

PROCEDURE ANALYSE examines the result from a query and returns an analysis of the results that suggests optimal data types for each column that may help reduce table sizes. Simply append PROCEDURE ANALYSE to the end of a select statement:

SELECT f1, f2 FROM t PROCEDURE ANALYSE(20, 2000);

where

• 10 is is the maximum number of distinct values which is checked for columns;
• 2000 is the maximum amount of memory which can be allocated per column while trying to find all distinct values.

This MySQL feature can be helpful after importing new data or for checking your existing tables to verify whether columns datatype was designed providently.

When designing partitions keep in mind that it will be required to have the number of partitions and open file limit system variable in harmony.

Thus each partition creates an overhead of extra two files (#p.myd and #p.myi). I.e. partition by hash of month will produce 2 * 12 months = 24 extra files. Not too bad for the default MySQL settings. However if there is plan for much more partitions, the open file limit has to be tuned respectively. Otherwise the error “mysql out of resources when opening file errcode 24″ may occur.

To change the number of file descriptors available to MySQL, you can set open-files=2048 (or to any other reasonable number) in the MySQL configuration file.

If table is not empty, there may be duplicated records. If so a regular alter query to create a new unique index will return an error like below.

ERROR 1062 (23000): Duplicate entry ’132653-47′ for key 1

However keyword IGNORE allows to force a new unique index to drop duplicated rows.

ALTER IGNORE TABLE `t` ADD UNIQUE INDEX `i` (`f1`, `f2`);

In this case the output will be something like this:

Query OK, 507 rows affected (0.02 sec)
Records: 507 Duplicates: 0 Warnings: 0

So simply using IGNORE helps to save time on writing a custom script to clean up duplicates.