Tuesday, July 30, 2013

Machine Configs for Bash Php & Apache


System wide configs for Bash, Php, and Apache (Php).

 We are vitalizing our systems and want to get to a standard config that will allow our single code base to run on many servers that may be utilizing other servers for services such as MySQL, Memcached, NFS shares and so on. The idea is that the server itself could hold the config necessary for it to communicate with its sibling systems.

Example Diagram:


Notice how the top section is web, middle is database, and bottom is storage.
Now each set of web servers could communicate with one or more databases and storage servers. But how do you let your code know which ones it should be using?

Here is where I am breaking off into Environment Variables!

   

Environment Variables.

By creating a single set of OS based environment variables, we can share the same config across all code on a server and standardize the setup for a new server that looks at other service providers.

Step 1: Profile Configs.

All login shell connections have a profile with "configs" that are loaded via /etc/profile.d (CentOS) or something similar.  I created a file in profile.d that will auto load with most things like executing a Php CLI script via login shell. Then it just takes a few tweaks for Apache and others. ( Environment Variables 1 2 )

/etc/profile.d/custom.config.sh
export MYVAR=this custom var
export MYVAR2=another custom var
 Now, to load this into Apache you first need to add
. /etc/profile.d/custom.config.sh
to your /etc/sysconfig/httpd file. This tells apache to source that file when restarting.
Next you need to tell Apache what to include into its SERVER variables. It is inclusive only, so you need to specify the variable for them to be included. I did this in the main /etc/httpd/conf/httpd.conf file, but you should be able to do it an any of the /etc/httpd/conf.d/files.

Example:

PassEnv MYVAR MYVAR2
Now, this will allow these variables to be passed to Apache Php scripts for use. Here is a Php script to test your output.

test.php

 <?php
echo "<pre>SERVER\n";
print_r($_SERVER);
echo "ENV\n";
print_r($_ENV);
exit;
 This should show something like this..
SERVER
Array
(
    [HTTP_AUTHORIZATION] =>
    [MYVAR] => this custom var
    [MYVAR2] => another custom var

    [HTTP_HOST] => www.bla.com
    [HTTP_USER_AGENT] => Mozilla/5.0 (Windows NT 6.1; WOW64; rv:22.0) Gecko/20100101 Firefox/22.0
    [HTTP_ACCEPT] => text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
    [HTTP_ACCEPT_LANGUAGE] => en-US,en;q=0.5
    [HTTP_ACCEPT_ENCODING] => gzip, deflate
    [HTTP_CONNECTION] => keep-alive
    [PATH] => /sbin:/usr/sbin:/bin:/usr/bin
    [SERVER_SIGNATURE] =>
Apache/2.2.15 (CentOS) Server at www.bla.com Port 80


    [SERVER_SOFTWARE] => Apache/2.2.15 (CentOS)
    [SERVER_NAME] => www.bla.com
    [SERVER_ADDR] => 10.128.1.100
    [SERVER_PORT] => 80
    [REMOTE_ADDR] => 10.128.1.58
    [DOCUMENT_ROOT] => /var/www/vhosts/bla/website/www/
    [SERVER_ADMIN] => admin@bla.com
    [SCRIPT_FILENAME] => /var/www/vhosts/bla/website/www/index.php
    [REMOTE_PORT] => 57829
    [GATEWAY_INTERFACE] => CGI/1.1
    [SERVER_PROTOCOL] => HTTP/1.1
    [REQUEST_METHOD] => GET
    [QUERY_STRING] =>
    [REQUEST_URI] => /
    [SCRIPT_NAME] => /index.php
    [PHP_SELF] => /index.php
    [REQUEST_TIME_FLOAT] => 1375211324.713
    [REQUEST_TIME] => 1375211324
)
ENV
Array
(
)
   

 Overriding the Variables per Virtual Host

Another awesome thing is that you may override the variables per virtual host on the machine. This will allow us to override the config for one V.H. using SetEnv which is part of Apache Module mod_env.

Example config:

# bla.com public
<VirtualHost *:80>
    DocumentRoot /var/www/vhosts/bla/website/www/
    ServerName www.bla.com
    ServerAlias bla.com
    ErrorLog logs/www.bla.com-error_log
#  CustomLog logs/www.bla.com-access_log combined
    <Directory /var/www/vhosts/bla/website/>
        AllowOverride All
    </Directory>
    SetEnv SITE_NAME bla.com
    SetEnv MYVAR 192.168.0.100
    SetEnv MYVAR 192.168.0.100

</VirtualHost>

BASH

Now some more details on how to get these variables into BASH and Crontab Php Scripts.
#!/bin/bash -l
This is how to use bash to act as a shell login (-l) which will load the profile configs and hence our custom environment variables.

Example BASH script to test variables

#!/bin/bash -l
set > test.sh.txt

CRONTAB

We are down to the end here, with slight wave of the hand we can pass the environment variable to php crontab scripts also. This is done by using the bash -l command to do a shell login call and using the -c with executes a command via bash.

Example CRONTAB

[root@web1 vhosts]# crontab -l
# *    *    *    *    *  command to execute
# .    .    .    .    .
# .    .    .    .    .
# .    .    .    .    .
# .    .    .    .    ...... day of week (0 - 6) (0 to 6 are Sunday to Saturday, or use names)
# .    .    .    ........... month (1 - 12)
# .    .    ................ day of month (1 - 31)
# .    ..................... hour (0 - 23)
# .......................... min (0 - 59)

* * * * * cd /var/www/vhosts/; bash -l -c ./test.php
* * * * * cd /var/www/vhosts/; ./test.sh

Example Php Script ./test.php

#!/usr/local/bin/php
<?php

file_put_contents('./test.php.txt',var_export($_SERVER,true));
 

 

mysql security using config editor

Here is a sweet tidbit for those who need to do mysql scripts often.
This is great for bash scripts.


mysql_config_editor — MySQL Configuration Utility
The mysql_config_editor utility (available as of MySQL 5.6.6) enables you to store authentication credentials in an encrypted login file named .mylogin.cnf. The file location is the %APPDATA%\MySQL directory on Windows and the current user's home directory on non-Windows systems. The file can be read later by MySQL client programs to obtain authentication credentials for connecting to MySQL Server.
Stores encrypted login information.
mysql_config_editor set --login-path=local --host=localhost --user=fooman --password  

Then just use login-path for your scripts...
 cat some-random-queries.sql | mysql --login-path=local

Tuesday, July 16, 2013

Obvious For Some - How to tail log for connections on port, CentOS?

This may be obvious to a lot of users out there, but I am making myself a note.
Mostly because I am more likely to remember if I make a note!



tail -f /var/log/messages  |grep "DPT=3306"
This tails the messages log for any connection to port 3306, typically MySQL.
It is a handy way to see what IPs are connecting to MySQL.

Monday, July 15, 2013

Backup/Copy a MySQL table to another table on same server.

So I need to update our table copies at work due to moving to InnoDB table from MyISAM, finally!

I have created a simple script thanks to Todd's MySQL Blog for some insights into transportable tablespaces and MySQL dev site on table copying InnoDB tables.  Trick here is to ensure that you can still copy MyISAM tables during the transition.

More Error handling/checking to come...

create file: copy_table.php

#!/usr/local/bin/php
<?php
define('_REPLICATION_USER', 'xxx');
define('_REPLICATION_PASS', 'xxx');

$db = new PDO('mysql:host='.$argv[1].';dbname=mysql', 'xxx', 'xxx');

$originalschema=$argv[2];
$originaltablename=$argv[3];
$copyedschema=$argv[4];
$copyedtablename=$argv[5];

//copy_table.php localhost originalschema originaltablename copyedschema copyedtablename


// check schema info
$engine = $db->query("select engine from information_schema.tables 
where table_schema = '$originalschema' AND  table_name = '$originaltablename'")->fetch(PDO::FETCH_COLUMN);
$engine2 = $db->query("select engine from information_schema.tables where table_schema = '$copyedschema' AND  table_name = '$copyedtablename'")->fetch(PDO::FETCH_COLUMN);


if($engine && $engine2!=$engine){
    echo "creating $copyedschema.$copyedtablename\n";
    // different table type for copy table, drop it like its HOT!
    $db->query( "DROP TABLE $copyedschema.$copyedtablename" );
    // create copy
    $db->query( "CREATE TABLE IF NOT EXISTS $copyedschema.$copyedtablename LIKE $originalschema.$originaltablename" );
    // make sure there is something in there.
    $db->query( "INSERT INTO $copyedschema.$copyedtablename SELECT * FROM $originalschema.$originaltablename LIMIT 1" );
}

switch ($engine){
//####################################################################################
case "InnoDB":
    echo "InnoDB:\n\n";
    if(file_exists("/var/lib/mysql/$originalschema/$originaltablename.ibd")){
        $db->query( "ALTER TABLE $copyedschema.$copyedtablename DISCARD TABLESPACE" );
        $db->query( "FLUSH TABLES $originalschema.$originaltablename FOR EXPORT" );

        # save data file.
        exec("cp -pf /var/lib/mysql/$originalschema/$originaltablename.ibd /var/lib/mysql/$copyedschema/$copyedtablename.ibd");

        $db->query( "UNLOCK TABLES" );
        $db->query( "ALTER TABLE $copyedschema.$copyedtablename IMPORT TABLESPACE" );
    } else {
        echo "InnoDB file does not exist.\n\n";;
    }

    break;
//####################################################################################
case "MyISAM":
    echo  "MyISAM:\n\n";
    if(file_exists("/var/lib/mysql/$originalschema/$originaltablename.MYI")){
        $db->query( "FLUSH TABLES $originalschema.$originaltablename WITH READ LOCK" );

        # save data files.
        exec("cp -pf /var/lib/mysql/$originalschema/$originaltablename.frm /var/lib/mysql/$copyedschema/$copyedtablename.frm");
        exec("cp -pf /var/lib/mysql/$originalschema/$originaltablename.MYI /var/lib/mysql/$copyedschema/$copyedtablename.MYI");
        exec("cp -pf /var/lib/mysql/$originalschema/$originaltablename.MYD /var/lib/mysql/$copyedschema/$copyedtablename.MYD");

        $db->query( "UNLOCK TABLES" );
        $db->query( "FLUSH TABLES $copyedschema.$copyedtablename" );
    } else {
        echo "MyISAM file does not exist.\n\n";;
    }

    break;
//####################################################################################
}

Make the file executable.
 

Usage:
copy_table.php localhost originalschema originaltablename copyedschema copyedtablename


 

Wednesday, July 3, 2013

Recursive copy of files overwriting existing files

Here is a trick I just learned, again. It is amazing the things you forget and come back to every so often.

 cp command, at least in CentOS, defaults with alias cp -i (interactive). To bypass this and run a copy overwriting existing file you need to use /bin/cp or unalias the cp command. I am unsure about this being the default in other linux distros.

/bin/cp -Rfu source /var/www/html/

or

unalias cp
cp -Rfu source /var/www/html/

-R: Recursively copy files from directory.
-f:  if an existing destination file cannot be opened, remove it and try again
-u: copy only when the SOURCE file is newer than the destination file or when the destination file is missing

Recursive Deletion of .svn directories


After some evil bad version control errors by some crazy lunatic people I had to clean up svn for a project. The problem was that the files were added to svn on two different checkouts. When trying to do an update on one after a commit on the other it freaked the hell out. So i needed to recheckout a clean version and then copy the new files over the top of the other files. This does have the problem of loosing code, but I am doing it for the production copy which should be the primary code.

Here is how I cleaned up the live copy so I could copy those changes over the top of the clean checkout without copying the svn files over the top also.

find . -type d -name .svn -exec rm -rf {} \; # linux command line

the . in the find means start from the current directory.  the d is for types of file equals directories. the .svn is the name of the directories and finally rm -rf is the command to execute on each found item.