Monitoring Satellite 5 with PCP (Performance Co-Pilot)

During some performance testing we have done, I have used PCP to monitor basic stats about Red Hat Satellite 5 (could be applied to Spacewalk). I was unable to make it sufficient, but maybe somebody could fix and enhance it. I have taken lots from lzap. First of all, install PCP (PostgreSQL and Apache PMDA lives in RHEL Optional repo as of now, in CentOS7 it seems to be directly in base repo):

subscription-manager repos --enable rhel-6-server-optional-rpms
yum -y install pcp pcp-pmda-postgresql pcp-pmda-apache
subscription-manager repos --disable rhel-6-server-optional-rpms

Now start services:

chkconfig pmcd on
chkconfig pmlogger on
service pmcd restart
service pmlogger restart

Install PostgreSQL and Apache monitoring plugins

cd /var/lib/pcp/pmdas/postgresql
./Install   # select "c(ollector)" when it asks
cd /var/lib/pcp/pmdas/apache
echo -e "<Location /server-status>\n  SetHandler server-status\n  Allow from all\n</Location>\nExtendedStatus On" >>/etc/httpd/conf/httpd.conf
service httpd restart
./Install
# Configure hot proc
cat >/var/lib/pcp/pmdas/proc/hotproc.conf <<EOF
> #pmdahotproc
> Version 1.0
> fname == "java" || fname == "httpd"
> EOF

And because I have Graphite/Grafana setup available, I was pumping selected metrices there (from RHEL6 which is with SysV):

# tail -n 1 /etc/rc.local
pcp2graphite --graphite-host carbon.example.com --prefix "pcp-jhutar." --host localhost - kernel.all.load mem.util.used mem.util.swapCached filesys.full network.interface.out.bytes network.interface.in.bytes disk.dm.read disk.dm.write apache.requests_per_sec apache.bytes_per_sec apache.busy_servers apache.idle_servers postgresql.stat.all_tables.idx_scan postgresql.stat.all_tables.seq_scan postgresql.stat.database.tup_inserted postgresql.stat.database.tup_returned postgresql.stat.database.tup_deleted postgresql.stat.database.tup_fetched postgresql.stat.database.tup_updated filesys.full hotproc.memory.rss &

Problems I had with this

For some reasons I have not investigated closely, after some time PostgreSQL data were not visible in Grafana. Also I was unable to get hotproc data available in Grafana. Also I was experimenting with PCP's emulation of Graphite and its Grafana, but PCP's Graphite lack filters which makes its usage hard and not practical for anything beyond simple stats.

DNS and "next-server" in DHCP configuration on libvirt's dnsmasq

I was playing with Satellite and re-provisioning client registered to it. This is awkward when you do it remotely and on real hardware - for me it is difficult to setup (if you want DNS and DHCP) and when client fails during re-provisioning, you either have to have physical access to it, or client have to have some kind of remote management console. Using libvirt is, on the other hand, very straightforward and you can get DNS and DHCP for free.

# virsh net-edit --network default
<network>
  <name>default</name>
  <uuid>970b7e2e-88d1-4100-8a2a-8db36c911d4c</uuid>
  <forward mode='nat'/>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:f1:e9:9a'/>
  <dns>
    <host ip='192.168.122.46'>
      <hostname>sat-emb.example.com</hostname>
    </host>
    <host ip='192.168.122.170'>
      <hostname>proxy.example.com</hostname>
    </host>
    <host ip='192.168.122.25'>
      <hostname>client.example.com</hostname>
    </host>
  </dns>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.254'/>
      <bootp file='/pxelinux.0' server='192.168.122.46'/>
    </dhcp>
  </ip>
</network>

<network><dns> configures hostname and their IPs form domain name resolution.

<network><ip><dhcp><bootp> allows me to set server which serves as PXE network boot server and file it clients should request. In my case, 192.168.122.46 is a Satellite with tftp running and configured.

NOTE: I have noticed that guests can not translate outer world hostnames to IPs - it looked like dnsmasq on the virtualization host is not forwarding requests it can not resolve to DNS servers from /etc/resolv.conf. Adding "<dns><forwarder addr="ip.of.another.nameserver" domain="internal.network.com"/>..." and restarting the network did not helped. At the end I have discovered there is forgotten no-resolv option in /var/lib/libvirt/dnsmasq/default.conf. When I have removed it and restarted network to regenerate config, it worked. I have probably forgotten it there in some previous adventures. From dnsmasq manual page:

       -R, --no-resolv
              Don't read /etc/resolv.conf. Get upstream servers
              only from the command line or the dnsmasq
              configuration file.

Creating "Solaris patch" for testing with Spacewalk

Spacewalk is Linux systems management solution. Until some time ago it had support for Solaris clients as well, but that was removed recently. This needs some testing right?

In Solaris, you have 3 ways to deliver software (AFAICT - I have absolutely 0 knowledge about administering Solaris):

• package - check OpenCSW project for some of these and if you wanted to push these to Spacewalk, you had to throw them to solaris2mpm utility which creates *.mpm package from these *.pkg.gz files. These *.mpm were then push-able to Spacewalk using rhnpush. When pushed, this package will appear in "Packages" tab of your Solaris channel.
• Solaris patch - this is AFAICT something created by SUN (or Oracle) only and only distributed by them via payed subscriptions, I have not found any guide on how to create one yourself. Again, you need to use solaris2mpm to transform the file to push-able *.mpm file.
• Solaris patch cluster - same as for Solaris patch

This post is about very lame way on how to create solaris patch file which after push appears under "Patches" tab in older Spacewalk in Solaris channel. There is absolutely no intention to have this actually installable by Solaris client.

1. First of all notice solaris2mpm is broken in Spacewalk so use version from Satellite I have not reported it, as the functionality was actually removed anyway.
2. Notice that solaris2mpm is present in normal fedora rhnpush package as well, but see this old bug about its missing dependencies
3. Install heirloom-pkgtools from Spacewalk's build system. Although this build (direct link to rpm) is very old, it worked on mine Fedora 25
4. Build aaa-1.pkg:

   $ pwd
   /tmp/solaris_yay
   $ cat pkginfo
   PKG=aaa-1
   NAME=Just a demo solaris patch
   VERSION=0.0.1
   CATEGORY=application
   DESC=Some loooong description of this cool package or patch or whatever
   ARCH=i386
   VENDOR=http://where.you.got.it
   EMAIL=root@ocalhost
   $ rm -rf aaa* README.*; P=aaa-1; mkdir $P; date > $P/data; echo "Date: $( date +%Y-%m-%d )" > README.$P; echo "Relevant Architectures: i386" >> README.$P
   $ (echo 'i pkginfo'; pkgproto /tmp/solaris_yay/README.aaa-1=/README.aaa-1 /tmp/solaris_yay/aaa-1=/) >prototype
   $ pkgmk -o -d /tmp/; echo $?; pkgtrans -s /tmp /tmp/aaa-1.pkg aaa-1; echo $?
   

   I have made these ugly lines because I have been experimenting a lot with that and it allowed me to kinda automate parts of the process
5. Now on RHEL6 Satellite (so solaris2mpm works - see first point here) with heirloom-pkgtools package installed (see third point) (so you do not need to do this on Solaris machine) run:

   # solaris2mpm aaa-1.pkg
   Writing patch-solaris-aaa-1-1.i386-solaris-patch.mpm
   

6. Push resulting *.mpm to the Spacewalk's/Satellite's Solaris channel using rhnpush and enjoy looking at "Patches" tab of the channel filled with some content without SUN/Oracle subscription.

Some links I have used along the way:

• Home of The Heirloom Packaging Tools, but this fork (?) might be handy
• Nice guide of How to make a Solaris package, alternate method
• Some info about Sun Solaris Application Packaging - patch mentioned there is not a "Solaris patch", but the link was quite important for me

Difference in Spacewalk's API and almost dirrect SQL performance

Imagine you want to get list of hosts registered to your Spacewalk, ideally with groups they are registered to and you want to do it repeatedly, so performance matters. Lets measure it.

I have Spacewalk 2.4 on a 2 CPU virtual system with 4 GB or RAM (Virtual, really? Not ideal for perf measurement, I know.) and I have created 1000 system profiles on it. There are 2 ways how to get the data out of the Server: command-line spacewalk-report inventory utility (needs to be run on a system running Spacewalk, queries directly the database) or system API (can be ran from anywhere, but data have to go from DB through spacewalk's Java stack and to XML which is then transferred to you over the network). API script to measure can look like this (well, this does not output obtained data):

#!/usr/bin/env python

import xmlrpclib
import time

server = xmlrpclib.Server('http://<fqdn>/rpc/api')
key = server.auth.login('<user>', '<pass>')
for i in range(100):
  before = time.time()
  systems = server.system.listUserSystems(key)
  for s in systems:
    detail = server.system.getNetwork(key, s['id'])
    groups = server.system.listGroups(key, s['id'])
  after = time.time()
  print "%s %s %s %s" % (len(systems), before, after, after-before)
server.auth.logout(key)

Here are mine results (averages from 100 repetitions performed directly after spacewalk-service restart):

method	average duration	note
spacewalk-report inventory	1.4 seconds	Needs to run directly on Spacewalk
API with system.listUserSystems() only	0.9 seconds	Provides systm ID and profile name only (does not equal to hostname)
API with system.listUserSystems() and system.getNetwork()	23.8 seconds	Gives you IP and hostname
API with system.listUserSystems() and system.getDetails()	27.5 seconds	Gives plenty of info, including hostname, but not groups
API with system.listUserSystems(), system.getNetwork() and system.listGroups()	52.4 seconds	Finally, ths one gathers hostname and system groups

So, depends on what you want to achieve and how often do you want to run the script. Also, in API script case, you have to keep login (or logins when you need to run for multiple organizations) somewhere. Fortunatelly you can use read-only API user for this.