Other aop/ssh

Table Of Contents

Previous topic

Trac+SVN backup/transfer

Next topic

Offline DB Backup

This Page

Daya Bay Links

Content Skeleton

SSH Setup For Automated transfers

The basics of setting up passwordless SSH are described in env:wiki:PasswordLessSSH

Debugging Blocked SSH

Daily transfers of large tarballs often fall foul of network blockages from institute network administrators. If SSH connections fail and pinging succeeds a possible cause is the blockage of port 22 from the web server by intermediate routers.

In order to check this try running an SSH daemon on another port and connect to that. For example, on the destination cms01.phys.ntu.edu.tw start sshd on port 1234 (may need to open the port on the firewall at destination):

[blyth@cms01 ~]$ sudo /usr/sbin/sshd -d -p 1234
Password:
main(5568) debug1: TOKEN IS afstokenpassing
...

This allows testing an ssh connection over a non-standard port:

[dayabay] /var/log > ssh -p 1234 -v -v -v cms01.phys.ntu.edu.tw
OpenSSH_4.3p2-6.cern-hpn, OpenSSL 0.9.7a Feb 19 2003
ssh(6369) debug1: Reading configuration data /home/blyth/.ssh/config
ssh(6369) debug1: Reading configuration data /etc/ssh/ssh_config
...
[blyth@cms01 ~]$

ssh-agent process monitoring

On nodes from which cron controlled daily backups to remote boxes are performed it is necessary to keep the ssh-agent process running. This requires manual steps to start and authenticate the agent following server reboots.

For example on dayabay.ihep.ac.cn the cron commandline for the blyth account:

21 14 * * * ( . $ENV_HOME/env.bash ; env- ; python- source ; ssh-- ; ssh--agent-monitor root ) > $CRONLOG_DIR/ssh--agent-monitor.log 2>&1

This performs a daily check with function ssh–agent-monitor root using pgrep to look for the ssh-agent process. If not found a notification email is sent, such as:

From: me@dayabay.ihep.ac.cn
Date: 19 July 2013 14:21:02 GMT+08:00
Subject: === ssh--agent-check-user : Fri Jul 19 14:21:02 CST 2013

From: me@localhost
To: blyth@hep1.phys.ntu.edu.tw

=== ssh--agent-check-user : Fri Jul 19 14:21:02 CST 2013
=== ssh--agent-check-user : ssh-agent for user root NOT FOUND

The remedy is to use ssh–agent-start which prompts for the ssh key passphrase in order to authenticate the restarted agent, and allow the passwordless transfer of backup tarballs to proceed.