Category Archives: Server Development

Lame Scripting Awards

Lame Scripting Awards

How many people have been using cat to open files in their shell only to pipe the output to another command, or how many have piped things to wc -l from grep? Don’t understand what I am talking about, and want to be enlightened? Check out: Useless use of cat award. This is an old article but the information is probably more relevant than some other information presented in class or online. Lets be honest though and realize that there is more than one way to skin a scripted cat.

Perl parsing field value pairs without separation

It is possible with perl functional programming to solve some interesting problems.

Suppose that you have a data set that has been processed to a degree where there was an error in the processing which removed some required delimiters such as the following:

some_data=some_valuesome_data=some_valuesomedata=somevalue
Continue reading →

Plesk RHEL 5.2 500 Error Virtualhosted Sites

Upon receiving a 500 error on multiple sites due to what appears to be an update to Plex followed by an update to RHEL5, we have found a way to track the issue early and resolve quickly. There is a script that will be posted at the end of this message.

Tracking the issue:

The first sign of trouble was the 500 error which appears in a virtualhosted server environment with more than one site failing, but also more then one site operating without issue. Since the operation of several sites were working this meant to us that the php, mysql and apache components were functioning normally. The 500 error hinted to us about cgi and perl possibly failing due to missing modules or linked headers. Also the failing sites appeared to be using Miva Merchant. Miva itself was not the cause of the issue however.

using the ldd command we were able to prove that all libraries were in the correct places and being found by ld.

ldd /usr/bin/mivavm
linux-gate.so.1 => (0x0056b000)
libm.so.6 => /lib/libm.so.6 (0x008c4000)
libdl.so.2 => /lib/libdl.so.2 (0x008ef000)
libpthread.so.0 => /lib/libpthread.so.0 (0x008f6000)
libc.so.6 => /lib/libc.so.6 (0x00769000)
/lib/ld-linux.so.2 (0x0074a000)

ldd /usr/bin/php-cgi
linux-gate.so.1 => (0x007c5000)
libcrypt.so.1 => /lib/libcrypt.so.1 (0x00412000)
libaspell.so.15 => /usr/lib/libaspell.so.15 (0x009e6000)
libpspell.so.15 => /usr/lib/libpspell.so.15 (0x00926000)
libgmp.so.3 => /usr/lib/sse2/libgmp.so.3 (0x009ae000)
libdb-4.3.so => /lib/libdb-4.3.so (0x02e91000)
libpthread.so.0 => /lib/libpthread.so.0 (0x008f6000)
libcurl.so.3 => /usr/lib/libcurl.so.3 (0x001fd000)
libbz2.so.1 => /usr/lib/libbz2.so.1 (0x02e7e000)
libz.so.1 => /usr/lib/libz.so.1 (0x00911000)
libpcre.so.0 => /lib/libpcre.so.0 (0x00cb9000)
libresolv.so.2 => /lib/libresolv.so.2 (0x00b60000)
libm.so.6 => /lib/libm.so.6 (0x008c4000)
libdl.so.2 => /lib/libdl.so.2 (0x008ef000)
libnsl.so.1 => /lib/libnsl.so.1 (0x00993000)
libxml2.so.2 => /usr/lib/libxml2.so.2 (0x00292000)
libgssapi_krb5.so.2 => /usr/lib/libgssapi_krb5.so.2 (0x001cd000)
libkrb5.so.3 => /usr/lib/libkrb5.so.3 (0x00d33000)
libk5crypto.so.3 => /usr/lib/libk5crypto.so.3 (0x00d0a000)
libcom_err.so.2 => /lib/libcom_err.so.2 (0x00cfa000)
libssl.so.6 => /lib/libssl.so.6 (0x00183000)
libcrypto.so.6 => /lib/libcrypto.so.6 (0x00b76000)
libidn.so.11 => /usr/lib/libidn.so.11 (0x00b14000)
libc.so.6 => /lib/libc.so.6 (0x00444000)
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x028a3000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00110000)
/lib/ld-linux.so.2 (0x0074a000)
libkrb5support.so.0 => /usr/lib/libkrb5support.so.0 (0x00cff000)
libkeyutils.so.1 => /lib/libkeyutils.so.1 (0x00de3000)
libselinux.so.1 => /lib/libselinux.so.1 (0x00931000)
libsepol.so.1 => /lib/libsepol.so.1 (0x0094b000)

Running the commands from the command line showed that they were completing fine and not segfaulting on as standard run. The environment for Miva is not set which explains the second line and this happens after the headers are sent which limits its cause.

/usr/bin/mivavm
Content-Type: text/html
Set-Cookie: htscallerid=
ec99e8351f3312599b5c4e1e0a78183b; expires=Thu, 28-Jun-2012 16:02:03 GMT; path=/

Configuration Error: Error loading configuration library: Unable to locate/load libmivaconfig.so
Please contact the server administrator: webmaster@localhost

/usr/bin/php-cgi

test#>

inspecting apache logs showed nothing but a restart a few days earlier.

[Tue Jun 28 22:39:22 2011] [notice] Apache configured — resuming normal operat ions

inspecting site specific logs from:

/var/www/vhosts/domain.com/statistics/logs/error_log

[Wed Jun 29 12:53:57 2011] [error] [client 10.0.0.151] suexec policy violation: see suexec log for more details
[Wed Jun 29 12:53:57 2011] [error] [client 10.0.0.151] Premature end of script headers: mivavm

inspecting suexec_log and suexec.log reveal the following:

lines in suexec_log
[2011-06-28 08:54:19]: uid: (#####/#####) gid: (2524/2524) cmd: mivavm

lines in suexec.log

[2011-06-28 08:59:34]: uid: (#####/#####) gid: (2524/2524) cmd: mivavm
[2011-06-28 08:59:34]: command not in docroot (/usr/bin/mivavm)

This hinted at an issue with suexec configuration of which docroot is precompiled in the command.
Upon inspecting /usr/local/psa/suexec/psa-suexec in vi, the file /usr/bin/minavm can be found
which hinted that this file is working and running the command yields no response which seems ok and no logs changed.

Running /usr/sbin/suexec and making note of the fact that it is old and that there are several versions gave the clue.

-r-s–x— 1 root root suexec
-r-s–x— 1 root apache suexec.saved_by_psa
-r-s–x— 1 root apache suexec.saved_by_psa.01.05;14:15
-r-s–x— 1 root apache suexec.saved_by_psa.01.06;18:03

Verified this assumtion with online post:

http://forums.theplanet.com/lofiversion/index.php/t57865.html

Modified the script to suit our environment

#!/usr/bin/bash
a1=`ls -la /usr/sbin/suexec | tr “-” ” ” | gawk ‘{print substr($0,1,56)}’`
a2=`ls -la /usr/local/suexec/psa-suexec | tr “-” ” ” | gawk ‘{print substr($0,1,56)}’`
if [[ “$a1” != “$a2″ ]]
then
cp /usr/sbin/suexec /usr/sbin/suexec-saved-$(date +%Y”-“%m”-“%d)
cp -Rfp /usr/local/suexec/psa-suexec /usr/sbin/suexec
fi

-r-s–x— 1 root apache 19964 Mar 31 00:05 suexec
-r-s–x— 1 root root 11608 Jun 29 13:12 suexec-saved-2011-06-29
-r-s–x— 1 root apache 21656 Dec 23 2009 suexec.saved_by_psa
-r-s–x— 1 root apache 20896 Dec 18 2007 suexec.saved_by_psa.01.05;14:15
-r-s–x— 1 root apache 20612 Apr 25 2008 suexec.saved_by_psa.01.06;18:03

Site resume normal operation

Mysql Script Tools

Mysql Script Tools

There are a number of operations in mysql that could benefit from being optimized in a script that performs these actions on all tables. A List of maintenance commands can be found. As it is, currently I have created a number of scripts for performing Analysis, Backup, Check, Checksum, Optimize, Repair, Restore. The tools are designed to operate on all table names accessible from the INFORMATION_SCHEMA.TABLES with the exception of the information_schema tables.

Continue reading →

Perl Socket.so and inet_aton.so

Upon building a module for perl I ran into a build issue with Socket.so in regards to an undefined symbol: inet_aton.so. When taking a closer look at the socket.so lib, it is apparent that the “proper” libraries should be there, however, in Solaris there are a few functions that do not exist in the same spots as other OS&squo;. So in order to properly build you need to add -lresolv or when you are invoking something that is prebuilt like cpan and using perl Socket.so you can simply use:

LD_PRELOAD=libresolv.so path_to_cpan

Managing logs with Sqlite and syslogd

Update Nov 26, 2010

Looking at the code for current version of syslogd from opensolaris. The exact sources can be obtained from this link which is what I will be working from as of tonight. The source code for sqlite can be obtained from sqlite.org, but I will not be getting into how to build that or anything but where I am injecting the library calls at this point. The current version of sqlite from the site is 3.7.3 and I will most likely not be using this version, but will be using a compatible one ( version 3 ). A look at this version of syslogd specific to solaris suggests a multithreaded architecture on a per file basis. We will however hopefully be using one file and possibly many tables or something to that effect. The entry points I have found thus far are in the conf_init function on line 3339 of syslogd.c and line 1396 of syslogd in the writemsg function. The conf_init function looks like a suitable sqlite init and open it is the location where all of the log files are opened. Line 5190 might be a possible location to close the sqlite database inside the reconfigure function and conf_init should bring it back up again. Line 1687 looks like the shutdown procedure location when not reconfiguring. The Schema will look like this: CREATE TABLE logs ( id  BIGINT AUTO, loglines TEXT, logtime TIMESTAMP, remotehost BIGINT ) ; or something similar ( this is not the actuall create statement but more of a psuedo create )

Managing logs with Sqlite and syslogd

I have often wondered why we as a society of techies create all these wonderful new technologies and fancy gadgets, but we never seem to get the logging thing right. I am sure that people have thought of using an interface to logging before, after all a text editor is a form of interface, sort of. Lets face it though, looking at log files through a console, xterm, a notepad editor or web interface that parses on the fly is just plain rediculous. I know that there are those who use automated tools to parse and split logs day by day an zip up the contents and back up the zip files and do all sorts of wonderful things, but I think we are missing a very important thing here. Why dont we try to encorporate the database straight into the logging system.

Benefits of database logging

There are many benefits to logging to a database, wether or not you are using a web interface.

  • The first being the sql query language which allows us to select a set of records.
  • The second, which is not any less important than the first is the ability to split the data into records and fields that are easily separated and parsed.
  • The third is most notable is the amount of data that you can put in a database without taking out the client.
  • The fourth is the ability to selectively retrieve, delete, manipulate.

Drawbacks

Allthough this may seem like the ultimate of ultimates to log to a database, there are some drawbacks even at first glance.

  • The first is the same as the last f=benefit, you can manipulate at will and effect a lot of data at once. This is not necessarily that that different from a regular file situation which you can merely delete, but more in the realm of corrupting or impersonating data in the form of date manipulation or otherwords
  • The System needs to be protected
  • There are potential problems with stability if modifying the core of syslog

Implementation

Although I have not actually made a patch or schema yet for this situation, I am looking into the code and plannng a project to implement a suitable patch to syslog whereas I can use sqlite as the database backend and syslogd as the front end. It may be possible to also do this with apache, postgres, mysql or other services that are not tied to syslog. The base framework would, however, be to simply get syslogd working correctly

Happy Coding and anyone interested in helping or suggesting on this project, drop us a line.

Using Perl to time Bash Calls – Bash fine grain timer

Using Perl to time Bash Calls – Bash fine grain timer

I was looking for a way to cross platform way to make a microsecond timer or anything less than one second for that matter in bash. Aside from doing a fork each time this function needs to be called, this is a pretty good alternative to c programming because it is quick and dirty. This function will allow us to be portable and not think too much about the implementation for the time being.

#!/usr/bin/perl
use strict;
use warnings;
use Time::HiRes qw(usleep nanosleep);
usleep(100);

Bash Multi Get using wget

Bash Multi-Get using wget

If you were ever looking at using wget with a proxy – then this is probably the script for you. I haven&squo;t spent much time testing this and it requires a helper program written in perl to get the timer resolution to handle properly. The idea behind this script is to be able to traverse a list of proxies so that you can test accessing your site from a bunch of “different places at once”.
Continue reading →

Bash SQL Query Helper

What is Bash SQL Query Helper?

This script is a very simple script that helps one to run sql commands from the command line with the command name and the query as opposed to running all sorts of switches. This is definately not a complicated script and there are probably some security considerations around using such a script with certain fields saved in it. But the main goal is to be able to call the mysql client to give me a result set as simple as possible.

Continue reading →

Simple Perl Template Engine

Simple Perl Template Engine

 

There are many uses for a template engine, but the basic idea to my knowledge is to reuse a design and replace the content quickly and efficiently. Like code re-usage, this helps to make our lives easier by allowing us to save time and preventing us from having to duplicate manually and create a page for every link. This also allows us to grab data from a data store, or database that will make managing content somewhat easier.

Allocate Version Numbers With Bash

Allocate Version Numbers With Bash

Bash at it&squo;s simplest form is a program that runs other programs for you based on input. Bash also has some other features than this such as the ability to store and retrieve variables stored in the Environment for a shell. If you invoke bash and then exit, you will loose the Environment that was set inside the script unless some special things are done.
Continue reading →