Tag Archives: sysadmin

Learn to rock Linux

You’ll have to take a look at the currently unmet business needs to help determine what will be the most benefit to the company.

Basic administration ideas

  • Monitoring: setup Nagios to monitor ping, disk space, cpu usage, on all network infrastructure
  • Monitoring: setup Cacti to monitor disk usage with historical graphs
  • Documentation: are all the server configurations and network information stored in an easily accessible, editable format (think wiki)? If not, roll a wiki instance on your server and document everything. Mediawiki is free, and okay. Atlassian Confluence is better, and not much more than free.

Monitoring and documentation are the two most critical administrative tools. These are the two pillars of an environment.

If you grew up as an IT guy on Windows, then you’ve been done a disservice. It’s okay, you can unlearn the evil. Here are some key differences that will help you become acclimated to Linux:

  • Troubleshooting: there’s always a log file, for every service and application in linux. Having a problem? Consult the log file. Can’t find the log file? Look in the startup script for the application.
  • Troubleshooting: learn strace. strace is the trace of system calls executed by a given program. You can see what files are opened, written to, etc, and this is a critically important tool for troubleshooting issues.
  • Troubleshooting: use “bash -x”. Running a bash script with the “-x” option shows you the execution line by line, and is very valuable for troubleshooting as well.
  • Troubleshooting (advanced): gdb (gnu debugger). You can run a binary program with gdb and inspect the call stack, see what’s stored in memory, and do other sweet shit.
  • Troubleshooting tools: netstat, ps, top, lsof, dig, nslookup, ping, tcpdump. Netstat is your eyes into what program is listening / bound to what port. ps and top let you know what programs are running and what their memory usage is. lsof lists open files, and tells you what programs are accessing them – very useful. dig, nslookup, ping, and tcpdump are all handy network troubleshooting tools.
  • Learning: man pages. Always read the man pages when you’re learning a new command, or try to. Sometimes they’re really very indiscernible, but often times they are not, and it’s faster than lazily googling for your quarry.
  • Learning: patience. Breathe. You can do it, but you’ll have to take it slow.
  • Learn what an inode is. That’s important.
  • Next level shit: learn Python. Another way to put this: learn linux, and Python, and get a few years (5-8+) under your belt – you’ll be earning 6 figures as a Unix engineer for some fancy fucking company.

Above all, remember: in Linux there’s almost always a semi-sane answer for what’s happening. You just have to know where to look. Log files, bash -x, strace, etc. The best way to learn is to setup some basic services and get your learn on. Try BIND, NFS, Samba CIFS, Nagios, Cacti, SSH via PKI, and then come back if you’ve done all that.

Oh, and turn off SELinux at first. That’s gonna be a real pain in the ass while you’re learning. Then turn it back on when you’re ready to do some security training.

Take some online (self-paced) redhat courses. These can be a little rote, but they’ve got value. Copy and paste the material into an electronic notebook for later use (http://evernote.com).

Hell if you can do all that, I’ve got a job for you.

Directory Services Command Line (dscl)

To make changes, all commands must be done with sudo privileges. To read only, no sudo is required:

Just as an LDAP server has a directory, every desktop and server computer has a local directory structure where the local user information is stored. This is most easily accessed using the dscl tool in command line. There are 2 modes for the tool, Interactive and Non-Interactive mode. Non-interactive mode is what a script (such as the hidden user script) would use, you type an entire command before hitting enter. Interactive mode gives you a dynamic environment to run in. (more…)

Xsan Maintenance

The proper Xsan volume maintenance is critical to the health of your filesystem. Short of a full crash with loss of configuration files, I have not seen any minor problems that this didn’t take care of (or at least help out tremendously). I try to do this to each volume at least once a month, and anytime there is a problem (i.e. filesystem crash). Start by stopping the volume: (more…)

Identify a file by inode

If you need to know what file a particular inode is associated with, you can use the following command:

find /Volumes/[volumename] -inum [number]

This can be useful in when used with the cvadmin command: “repof”┬áto get a report of open files. You may need to do this if you find that a particular command will not execute due to open files.

Configure Remote Desktop From the Command Line

All commands must be done with sudo privileges:

You can enable remote management with the these commands in command line. This is very useful after you have created a hidden admin user, because the GUI will not allow you to see the hidden user in the sharing preference pane. Its also very useful for scripting. I made a script that creates a hidden admin user and performs this configuration, and I rolled it into an OSX installer packager. Then I carry it around on my USB stick for any machines in my area that are not accepting ARD connections.


Creating a Hidden Admin User

This is a quick tutorial on some of the basics of creating a hidden user. This tutorial is the information that I used to create my Administrative Install Package that I put on all machines that I service. At a workplace people get very possessive of their machines (even though its not really “theirs”). Many of my users didn’t like seeing another username in the login screen. But I can’t service a hundred computers when I have no access into them. This way, I have full access and none of the users know. This is specific to Leopard, and I believe this also works with Snow Leopard. However, Tiger uses the netinfo command line instead, which is similar in its interactions, but not entirely the same.