Linux

From Mosuma
Jump to: navigation, search

bash

Bash (v3 and lower) sucks big-time in handling big arrays because

Just try reading in a huge file as an array and iterating through it

#!/bin/bash
FILES=path_to_long_file
IFS=$'\r\n' GLOBIGNORE='*' command eval 'ALONG=($(cat $FILES))'
N=${#ALONG[@]}
# Each of the following TAKES FOREVER for N=13m
regex=${ALONG[@]:0:5}
${ALONG[@]}

Configuration Management Tools

  • Salt
    • Python
  • Puppet
    • Ruby
  • Chef
  • CFengine
  • KickStart

Forcing reboot

Based on http://major.io/2009/01/29/linux-emergency-reboot-or-shutdown-with-magic-commands/

echo 1 > /proc/sys/kernel/sysrq 
echo b > /proc/sysrq-trigger

HAProxy routing by domain name

Ubuntu 12.04

Section "Screen"
     Identifier     "Default Screen"
     DefaultDepth   24
     SubSection "Display"
          Viewport   0 0
          Virtual   1920 1968
          Depth     24
     EndSubSection
EndSection

Section "Module"
     Load "glx"
EndSection


Openstack cloud platform on Ubuntu Linux 12.04

RAID on Linux

mdadm --create /dev/md1 --level=1 --raid-devices=2 /dev/sdb1 missing
mdadm -C /dev/md1 --level=raid1 --raid-devices=2 /dev/xvdc /dev/xvdd
mdadm --detail --scan >> /etc/mdadm/mdadm.conf
  • make sure there is one and only one line of RAID config in /etc/mdadm/mdadm.conf, otherwise the system will not reboot.
  • To stop raid, you must stop the underlying LVM lv and vg:
lvchange -an /dev/{vgname}/{lvname}
vgchange -an /dev/{vgname}
mdadm -S /dev/md0
  • Resize a /dev/mdX partition
mdadm --grow /dev/md1 --size=max
  • Resize a PV
pvresize /dev/md1


  • Assembling a RAID array from a snapshot on amazon EC2
mdadm --assemble /dev/md0
mdadm /dev/md0 --run
mdadm --add /dev/md0 /dev/xvdg
mdadm --detail --scan >> /etc/mdadm/mdadm.conf

Evolution and Exchange 2010

http://www.petenetlive.com/KB/Article/0000378.htm

Open source vector software on Linux

SSD and Linux LVM

--Zhangguiyu (talk) 07:06, 21 November 2012 (UTC)

$ lsblk -D

Look for the line that has your drive, then look below that to the LVM name. If you see "0B" in both the DISC-GRAN and DISC-MAX columns then discard is not passing through to the LVM partition. If you see something besides 0B then discard is enabled.


--Zhangguiyu (talk) 05:55, 28 June 2012 (UTC)

moving away from a broken PV

http://ds9a.nl/lvm-howto/HOWTO/cvs/lvm-howto/output/lvm-howto-6.html

Open Source Router

Configuring Postfix

--Zhangguiyu (talk) 02:46, 11 October 2014 (SGT)

Assumptions: Debian Linux

DNS resolution in postfix chroot jail

if the postfix server has multiple IP addresses with some isolated from the Internet, the postfix chroot jail may not have properly configured DNS resolution. In my example, I had two interfaces:

root@debian:/etc# ifconfig
eth0      Link encap:Ethernet  HWaddr 08:00:27:25:6c:f3  
          inet addr:192.168.56.102  Bcast:192.168.56.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe25:6cf3/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:73373 errors:0 dropped:0 overruns:0 frame:0
          TX packets:51265 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:6266810 (5.9 MiB)  TX bytes:10541875 (10.0 MiB)

eth1      Link encap:Ethernet  HWaddr 08:00:27:7e:10:5f  
          inet addr:10.0.3.15  Bcast:10.0.3.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe7e:105f/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:5625 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3701 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:1070910 (1.0 MiB)  TX bytes:345201 (337.1 KiB)

where

  • eth0 is a host-only network with no connection to the outside world
  • eth1 is a NAT network that interacts with the LAN on 192.168.0.xxx

my /etc/resolve.conf has the LAN DNS resolution:

#/etc/resolve.conf
nameserver 192.168.0.254

however, when this /etc/resolv.conf is copied over to the postfix chroot jail of /var/spool/postfix/etc/resolv.conf, it fails to resolv because the chroot jail has no NAT, and thus no access to the LAN DNS server 192.168.0.254.

Solutions

Solution 1: add google global DNS to /etc/resolv.conf or /etc/resolvconf/* (to dynamically generate /etc/resolv.conf) the following:

#/etc/resolv.conf
nameserver 192.168.0.254
nameserver 8.8.8.8

which will be copied over to /var/spool/postfix/etc/resolv.conf

and now postfix should be able to deliver mail to the outside world.

Solution 2: apt-get install resolvconf, then add this file to /etc/resolvconf/resolv.conf.d

#/etc/resolvconf/resolv.conf.d
# Google's DNS server accessible from any NAT network
nameserver 8.8.8.8

then restart postfix, and you will see the new /var/spool/postfix/etc/resolv.conf containing your added global DNS server

Solution 3: Under Docker

The above solutions 1 and 2 do not work under docker due to the lock down of dockerinstance:/etc/resolve.conf as a read-only link, see https://github.com/docker/docker/issues/1951

Thus the easiest solution is to add the globally accessible DNS when running docker:

docker run --dns 8.8.8.8 -p 80:80 -p 23:23 --cap-add SYS_PTRACE --name yourdockerimagename -i -t privaterepository/debian:7.5tagname

switching to use Maildir

Assumption: you use dovecot, procmail (for local delivery), postfix

Reference: https://wiki.debian.org/MaildirConfiguration A number of steps are needed, with the most IMPORTANT ones listed first:

  • add a line "home_mailbox = Maildir/" to /etc/postfix/main.cf
  • create a file /etc/procmailrc with the following content
MAILDIR=$HOME/Maildir/
DEFAULT=$MAILDIR
VERBOSE=no
LOGFILE=$MAILDIR/log

At this point, the system will deliver mail to the maildir for local users, but root user still has mail delivered to /var/spool/mail, which must be fixed with the next step:

  • (TO BE DEPRECATED, and no effect for root user) edit /etc/login.defs with
# /etc/login.defs
MAIL_DIR      Maildir/
#MAIL_DIR        /var/mail
# Note: QMAIL_DIR alone does not work as advertised... also need MAIL_FILE
MAIL_FILE      Maildir/
  • (NEEDED for root user Maildir to work) create this file /etc/profile.d/maildir.sh
#!/bin/sh
# see https://wiki.debian.org/MaildirConfiguration
#
# needed to override /etc/pam.d/login which set MAIL variable to /var/spool/*
export MAIL=~/Maildir
  • (NEEDED For internet email delivery to Maildir) edit /etc/dovecot/conf.d/10-mail.conf
# /etc/dovecot/conf.d/10-mail.conf 
#mail_location = mbox:~/mail:INBOX=/var/mail/%u
mail_location = maildir:~/Maildir
  • (OPTIONAL)
# /etc/dovecot/conf.d/10-mail.conf 
# When copying a message, do it with hard links whenever possible. This makes
# the performance much better, and it's unlikely to have any side effects.
maildir_copy_with_hardlinks = yes

# Assume Dovecot is the only MUA accessing Maildir: Scan cur/ directory only
# when its mtime changes unexpectedly or when we can't find the mail otherwise.
maildir_very_dirty_syncs = no

SASL configuration

Reference: http://www.howtoforge.com/postfix-dovecot-warning-sasl-connect-to-private-auth-failed-no-such-file-or-directory

For this type of error in /var/log/mail.err

debian postfix/smtpd[3631]: fatal: no SASL authentication mechanisms

fix it by

  • adding the following "unix_listener" fragment
  # Postfix smtp-auth
  unix_listener /var/spool/postfix/private/auth {
    mode = 0666
    user = postfix
    group = postfix
  }

to the "service auth" section of /etc/postfix/conf.d/10-master.conf. The following shows that section *after* the addition.

service auth {
  # auth_socket_path points to this userdb socket by default. It's typically
  # used by dovecot-lda, doveadm, possibly imap process, etc. Users that have
  # full permissions to this socket are able to get a list of all usernames and
  # get the results of everyone's userdb lookups.
  #
  # The default 0666 mode allows anyone to connect to the socket, but the
  # userdb lookups will succeed only if the userdb returns an "uid" field that
  # matches the caller process's UID. Also if caller's uid or gid matches the
  # socket's uid or gid the lookup succeeds. Anything else causes a failure.
  #
  # To give the caller full permissions to lookup all users, set the mode to
  # something else than 0666 and Dovecot lets the kernel enforce the
  # permissions (e.g. 0777 allows everyone full permissions).
  unix_listener auth-userdb {
    #mode = 0666
    #user = 
    #group = 
  }

  # Postfix smtp-auth
  unix_listener /var/spool/postfix/private/auth {
    mode = 0666
    user = postfix
    group = postfix
  }

  # Auth process is run as this user.
  #user = $default_internal_user
}
  • restart dovecot and check that the unix socket file has been created:
root@debian:/etc/dovecot/conf.d# /etc/init.d/dovecot restart
[ ok ] Restarting IMAP/POP3 mail server: dovecot.
root@debian:/etc/dovecot/conf.d# ll /var/spool/postfix/private/auth 
srw-rw-rw- 1 postfix postfix 0 Oct 11 02:30 /var/spool/postfix/private/auth
root@debian:/etc/dovecot/conf.d#

Misc

  • Using you wired ethernet connected ubuntu notebook as a Wifi-AP
    • Original instructions
    • Discussion
    • Note that you need to install a dhcp server *and* configure it as mentioned in the discussion above
      sudo apt-get install isc-dhcp-server