Monitoring Satellite 5 with PCP (Performance Co-Pilot)

During some performance testing we have done, I have used PCP to monitor basic stats about Red Hat Satellite 5 (could be applied to Spacewalk). I was unable to make it sufficient, but maybe somebody could fix and enhance it. I have taken lots from lzap. First of all, install PCP (PostgreSQL and Apache PMDA lives in RHEL Optional repo as of now, in CentOS7 it seems to be directly in base repo):

subscription-manager repos --enable rhel-6-server-optional-rpms
yum -y install pcp pcp-pmda-postgresql pcp-pmda-apache
subscription-manager repos --disable rhel-6-server-optional-rpms

Now start services:

chkconfig pmcd on
chkconfig pmlogger on
service pmcd restart
service pmlogger restart

Install PostgreSQL and Apache monitoring plugins

cd /var/lib/pcp/pmdas/postgresql
./Install   # select "c(ollector)" when it asks
cd /var/lib/pcp/pmdas/apache
echo -e "<Location /server-status>\n  SetHandler server-status\n  Allow from all\n</Location>\nExtendedStatus On" >>/etc/httpd/conf/httpd.conf
service httpd restart
./Install
# Configure hot proc
cat >/var/lib/pcp/pmdas/proc/hotproc.conf <<EOF
> #pmdahotproc
> Version 1.0
> fname == "java" || fname == "httpd"
> EOF

And because I have Graphite/Grafana setup available, I was pumping selected metrices there (from RHEL6 which is with SysV):

# tail -n 1 /etc/rc.local
pcp2graphite --graphite-host carbon.example.com --prefix "pcp-jhutar." --host localhost - kernel.all.load mem.util.used mem.util.swapCached filesys.full network.interface.out.bytes network.interface.in.bytes disk.dm.read disk.dm.write apache.requests_per_sec apache.bytes_per_sec apache.busy_servers apache.idle_servers postgresql.stat.all_tables.idx_scan postgresql.stat.all_tables.seq_scan postgresql.stat.database.tup_inserted postgresql.stat.database.tup_returned postgresql.stat.database.tup_deleted postgresql.stat.database.tup_fetched postgresql.stat.database.tup_updated filesys.full hotproc.memory.rss &

Problems I had with this

For some reasons I have not investigated closely, after some time PostgreSQL data were not visible in Grafana. Also I was unable to get hotproc data available in Grafana. Also I was experimenting with PCP's emulation of Graphite and its Grafana, but PCP's Graphite lack filters which makes its usage hard and not practical for anything beyond simple stats.