yum shell - bat out of dependency hell

There's evil in the air and there's thunder in sky

(Meatloaf "Bat out of hell")

# yum install foo
[..]
Error: foo conflicts with bar

Again I have had the pleasure of having dependencies between RPM-packages ending my attempt to install a single package with a suggestion of removing core packages. I think this most often happen with Mysql or Percona packages, but I am sure MariaDB will be able to give you the same situation too. It's not the first time I have been here..

[root@ftp01-prod ~]# yum install Percona-Server-client-57
Loaded plugins: fastestmirror, priorities
Loading mirror speeds from cached hostfile
Resolving Dependencies
--> Running transaction check
---> Package Percona-Server-client-57.x86_64 0:5.7.20-19.1.el7 will be installed
--> Processing Dependency: Percona-Server-shared-57 for package: Percona-Server-client-57-5.7.20-19.1.el7.x86_64
--> Running transaction check
---> Package Percona-Server-shared-57.x86_64 0:5.7.20-19.1.el7 will be installed
--> Processing Dependency: Percona-Server-shared-compat-57 for package: Percona-Server-shared-57-5.7.20-19.1.el7.x86_64
--> Running transaction check
---> Package Percona-Server-shared-compat-57.x86_64 0:5.7.20-19.1.el7 will be installed
--> Processing Conflict: Percona-Server-shared-compat-57-5.7.20-19.1.el7.x86_64 conflicts Percona-Server-shared-56
--> Finished Dependency Resolution
Error: Percona-Server-shared-compat-57 conflicts with Percona-Server-shared-56-5.6.38-rel83.0.el7.x86_64
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest
[root@ftp01-prod ~]#

So if I want to install Percona-Server-client-57 I have to install Percona-Server-shared-compat-57 too, and that I can't because of the already installed Percona-Server-shared-56. OK, so I will just remove Percona-Server-shared-56 and then install Percona-Server-shared-compat-57 before doing the install I first tried to do:

[root@ftp01-prod ~]# yum remove Percona-Server-shared-56
[..]
Dependencies Resolved

================================================================================================================
 Package                           Arch            Version                      Repository                 Size
================================================================================================================
Removing:
 Percona-Server-shared-56          x86_64          5.6.38-rel83.0.el7           @percona-release          3.4 M
Removing for dependencies:
 MySQL-python                      x86_64          1.2.5-1.el7                  @centos_os                284 k
 fail2ban                          noarch          0.9.7-1.el7                  @epel                     0.0
 fail2ban-sendmail                 noarch          0.9.7-1.el7                  @epel                      11 k
 perl-DBD-MySQL                    x86_64          4.023-5.el7                  @centos_os                323 k
 postfix                           x86_64          2:2.10.1-6.el7               @centos_os                 12 M
 redhat-lsb-core                   x86_64          4.1-27.el7.centos.1          @anaconda                  45 k

Transaction Summary
================================================================================================================
Remove  1 Package (+6 Dependent packages)

Installed size: 16 M
Is this ok [y/N]:

Very much not OK. I'd like to at least keep things with descriptions like this

Description : The Linux Standard Base (LSB) Core module support
: provides the fundamental system interfaces, libraries,
: and runtime environment upon which all conforming
: applications and libraries depend.

The problem seems to be that Postfix needs libmysqlclient.so.18 which is provided by both Percona-Server-shared-56 and Percona-Server-shared-compat-57. So I just need to swap the former for the latter, and then I can run my original install.

OK, so I both want to remove a package and install a package. And I want to do it at the same time, so that I don't have to remove things like redhat-lsb-core. Did you notice the use of the word transaction in "Transaction Summary" from yum? A transaction is actually what I want. Luckily yum provides a way of doing this, and have probably done since forever, but I didn't learn about it till today. And as so many times before, it is a shell that solves our problems:

[root@ftp01-prod ~]# yum shell

Loaded plugins: fastestmirror, priorities

> remove Percona-Server-shared-56

> install Percona-Server-shared-compat-57

Loading mirror speeds from cached hostfile

> run

--> Running transaction check

---> Package Percona-Server-shared-56.x86_64 0:5.6.38-rel83.0.el7 will be erased

---> Package Percona-Server-shared-compat-57.x86_64 0:5.7.20-19.1.el7 will be installed

--> Finished Dependency Resolution

==================================================================================================

 Package                             Arch       Version                Repository            Size

==================================================================================================

Installing:

 Percona-Server-shared-compat-57     x86_64     5.7.20-19.1.el7        percona-release      1.2 M

Removing:

 Percona-Server-shared-56            x86_64     5.6.38-rel83.0.el7     @percona-release     3.4 M

Transaction Summary

==================================================================================================

Install  1 Package

Remove   1 Package

Total download size: 1.2 M

Is this ok [y/d/N]:

Yes, very much thank you! And then finally:

[root@ftp01-prod ~]# yum install Percona-Server-client-57

[..]

Dependencies Resolved

==================================================================================================

 Package                        Arch         Version                  Repository             Size

==================================================================================================

Installing:

 Percona-Server-client-57       x86_64       5.7.20-19.1.el7          percona-release       7.2 M

Installing for dependencies:

 Percona-Server-shared-57       x86_64       5.7.20-19.1.el7          percona-release       747 k

Transaction Summary

==================================================================================================

Install  1 Package (+1 Dependent package)

Total download size: 7.9 M

Installed size: 41 M

Is this ok [y/d/N]:y

[..]

Complete!

Done and done :-)

Copying permissions and ownership of files

One of the big joys I find in working with Linux and Unix systems is that there is always something new I can learn, even with tools I have been using for over 15 years.

Today I have been working for one of my customers on a script that for a bunch of files matching a glob will read each file, process it, and generate some output into a new file.

I need the output files to have the same permissions and ownership as their respective sources, and started to look into different more or less elaborate ways of doing that. But then it turns out that this is a very well solved problem already. Both chmod and chown have an option for this:

--reference=RFILE

              use RFILE's owner and group rather than specifying OWNER:GROUP values

So in my script have added two lines, and now the permissions and ownership are copied onto the new files

    chmod ${DSTDIR}/${LOGFILE} --reference=${LOGFILE}

    chown ${DSTDIR}/${LOGFILE} --reference=${LOGFILE}

I guess these options have been there for years and years, waiting for me:-)

When the radius server came to a grinding halt - and how we brought it back

One of my customers have a few hundred pieces of hardware spread around the country that are authenticated and get their IP addresses from a radius server (http://freeradius.org/) we run. In between the few hundred pieces of hardware and the radius server there is another server that among other things handles the lookups against the radius server  and then gives the clients their connection information.

This have been running for four years with no problems what so ever, handeling a handful or so authentications a minute.

Yesterday the intermediate server suddenly booted (we learned this later), and that led to a need reinitialize the network connections for all the little pieces of hardware (thus authenticating them and looking up their IPs) - all at once. The load on the radius server went through the roof, and we had large amounts of timeouts on the lookups and the radius server came to a grinding halt. Most of the little pieces of hardware didn't get their IP's and the customer's monitoring system went all red and no-one was enjoying them selves much.

A trivial radius server should be able to handle at least a few thousand requests a second, so we were a bit boggled by this. A restart of the service didn't help.

We saw log lines like these in large amounts

Thu Jan  6 15:30:26 2011 : Error: Discarding duplicate request from client FOOFOO3:49910 - ID: 32 due to unfinished request 895

Thu Jan  6 15:30:26 2011 : Error: WARNING: Unresponsive child (id 1314167728) for request 890

The server was also exhausted CPU-wise - and mostly spent it's time in system CPU.

This led me to suspect that the problem was related to some kind of busy waiting on some kind of resource. A quick check in the trusty old /proc filesystem showed that the radius process had a couple of hundred file descriptors pointing to the same file - /var/log/radius/radutmp - an 89MB file with almost 1M short lines.

So what is this radutmp file for? It is updated whenever a client logs in or out, and is used by the command  radwho to see who is logged in at the moment. On this system we have no need for that so I removed the relevant sections from the config and restarted the service - and service was immediately restored.

According to the default and heavily commented config file the radutmp file is not a log file, and does not need to be rotated. But ours have obviously been growing over the last years, so some kind of housekeeping should probably have been done. Out of curiosity I tried starting with an empty radutmp and the original config - and it seemed that the size of the radutmp file affects how hard it is for the server to do it's work  - which makes sense. (but this testing was not done in any way scietifically, and might just be misleading).

I think the main lesson to learn from this is that /proc is your friend - always. Secondary, remove parts of services you don't need - problems might just as well show up there.