A Small List of Best Practices for Sysadmins and Programmers

I present you with my small list of best practices that I have used in the past as reference material and guidelines for work I have done.  I hope that by sharing this it will inspire you to share your favorites, or even create your own best practices documents to help move the practice of system administration and programming forward.

Programming

  • Some general common-sense best practices for scripting.  It has a UNIX slant but the concepts can easily be translated to any system.  Use comments, name variables well, etc.  Generally, pretend like your code will be on public display. – http://www.computerworld.com/s/article/9180590/Scripting_best_practices
  • The use of “skeleton projects” always gives me mixed feelings.  On the one hand I very much agree with not inventing the wheel over and over again.  If you have something that is solid and well documented than why change it.  In my opinion skeleton projects should be well documented and well understood, lest they quickly turn into “cargo code” – http://reinout.vanrees.org/weblog/2010/07/30/skeleton.html
  • Not so much a list of best practices, but a list of what not to do.  The Anti Patterns Catalog is some enjoyable reading because, if your ego can stand it, you will recognize a lot of these.  By the way c2.com is the original wiki, yes seriously!

Web

  • Yahoo! has created a great checklist for speeding up your website.  The think I enjoy about this guide is that they are practical examples along with why you want to do it. If you like this, please check out their automated tool YSlow.

DNS

  • Monkey House has a nice Top 10 DNS best practices.  While, I’m not sure of this are truly the top ten most common slip ups or the top ten most important things to consider, it is a well-rounded look at some things you should consider when setting up or maintaining DNS
  • Again, not so much a list of best practices, but Cisco offers a great overview/refresher to dns as well as a detailed list for detecting and preventing dns abuse of Cisco products
  • ComputerWorld presents a decent, high-level overview of best practices for reliable dns and dhcp.  Referenced in the article is CERT’s  Securing an Internet Name Server.  Some of the techniques in this paper were specific to certain versions of BIND, but the theory can be applied elsewhere

Active Directory

Infosec

Thanks for reading.  I look forward to hearing what your best practices are.  Are they hard-won lessons learned on the job?  Industry standards?  Personal preferences?

A Totally Useless Port Scanner To Teach You Some Python

In case you are viewing this in a RSS reader: please refer to the following “gists” the scanner code and the output to follow along with the post.

I would like to present you a quick post to demonstrate the power of Python in a succinct little port scanner. The time taken from idea to finished product was about three minutes.

First an example of the program. Everything (hosts to check and the port range) is hard-coded into the script.

Not too powerful and not too quick, but it does the job.

Now let’s go through the code and explain each line. As a boss of mine used to say “break it down ‘Barney style‘”


#!/usr/bin/python

This is our “hashbang” line. Please refer to wikipedia on this one as its a little out of scope.


from socket import socket
import time

Python, being “batteries included“, is doing a lot of sophisticated work for us behind the scenes: opening and closing ports, managing memory, etc. Some super smart people have bundled up all of their expert work into packages and modules. For all of their hard work, all they ask of you is that you nicely call upon their code before you use it.

In the above lines we are “pulling” in the code for sockets (python’s way of communicating on a network) and time.


HOSTS = ['192.168.1.1', '192.168.1.2']
PORT_RANGE = [1,100]

As you might guess from this section, this is where we are configuring our variables. All caps is not necessary, I just chose it to stand out. Note that both variables (HOSTS and PORT_RANGE) are lists. A list is just a container with separate compartments for holding different pieces of data. The cool thing about lists is that they provide us a way to iterate (or walk-through) each piece (element) of the data.


for host in HOSTS:

This is pretty easy, all it says is iterate over each element in HOSTS. Every time you hit a new element set host to that value then execute all the code below, once you reach the end, start back at the top, setting host to the next value.

By the way, the indentation in Python is important. Python looks “down the page”, everything it sees that is indented at least one tab, it knows is part of the for loop. At the end of the loop, the blank space tells it to stop looking. These levels of loops (or functions, or conditions) and their indented commands can be embedded within each other, as seen in our example.


t1 = time.time()

We are setting the variable t1 to the current time. This way we have a reference point to figure out how long the scan took. Note again that since we are in a for loop, once we are finished, all of these variables get “reset” when we do our next iteration.


print "Testing %s" % host

In this line we are telling python to print whatever is enclosed in the double-quotes to the string. The ‘%s’ tells python, “hey man I’m going to print *something* here, not sure what it is, but I can tell it will be a string”. The ‘%’ just says that everything after this shouldn’t be printed, but it should be sent back to the print statement. If you have more than one ‘%s’ (or other types, see http://diveintopython.org/native_data_types/formatting_strings.html) you need to do it a little different. It might look something like this:

print "This is the first variable: %s, this is the second: %s" % (one, two)

Now we are getting into the meat of the program, the actual commands that do the port testing.


for port in range(PORT_RANGE[0],PORT_RANGE[1]+1):

You remember our for command, right? In this situation we are using the range command rather than a pre-built list. Effectively, range builds this for us. We simply give it the first port and the last port plus one.


s = socket()

This command simply makes s an instance of whatever socket() is. In this case socket() is a “socket object”, which is a bundle of that wonderful “batteries included” code we get from importing at the top. Now, instead of saying socket.something(), we can say s.something. The advantage of this is that we can also have r = socket() and when we call on r it lives in its own little world, unaffected by whatever s is up to.


if not s.connect_ex((host, port)):

This might be the most complex part of our program. But once we break it down, you will see it is just a few easy concepts chained together. Let’s start from the inside out: what are host and port?

In the first pass through our code, host would be set to 192.168.1.1 and port would be 1. Makes sense right? The host and port are enclosed in a structure called a “tuple”. A tuple is very similar to our lists from earlier, it is a little lighter-weight. This gives us a handy way to send the host, port pair as one object.

Where is this object going? We are sending this as input to the connect_ex, this is a function which is part of the socket that we have set to s. Okay, so if you have this all straight (s is a reference to the socketcode pulled in from our import, we are feeding the tuple of (host, port) to the connect_ex function) you should be asking yourself, “What the heck does connect_ex do?”. I will do better then tell you, I will show you.

The wonderful thing about Python is the amazing depth of documentation. So lets say we want to know what a socket does? We simply look it up in the Python documentation. Starting at http://docs.python.org/ we see there are many categories of documentation. What we are interested in now are the “batteries” that are included with Python, also known as the “Standard Library”. We find that here. Also as you grow more curious about the basic of Python, there is wonderful documentation on the syntax and pieces that make up the Python language.

Okay, so what does socket_ex do? Lets look it up in the documentation. As you can see socket_ex takes an address, which is our (host,port). The function will try to connect to the host and port specified and report if it was successful or not. If the connection is successful it will return True, otherwise it will return False.

Next we see the if part of the statement. All this is saying is if the result of socket_ex is 0 (which is a successful connection) execute the commands below. The reason for using 0 (or other error codes if the connection is not successful) is because the socket functions mirror the underlying C code, which is rather low-level. Feel free to read it over.

Now we see that if the connection was successful we are going to print out a statement. The only thing new here is the \t – this tells print to insert a “tab”.

You will see the next line: t2 = time.time() has jumped back a few levels of indentation. If you are unfamiliar with Python’s rules on indentation or just need to brush up, feel free to check out the Python Style Guide. What we are doing here is “jumping” out of the if statement and continuing our normal program flow. Now that we have tested all of the ports for the host, we are going to save the current time and in the next line print "Took %0.3f sec" % (t2 - t1) we are displaying how long the execution took.

You will notice that the print statement we are using a slightly different syntax than before. We are simply printing the time as a float with three significant digits. Again, review the print formatting if you would like to see all of the various ways to format your output.

And that’s it. The loop will repeat with the next host in our list and continue the operation again.

Admittedly, this is a very simple port scanner, meant as an instructional device. We make no attempts to verify or do any error checking. We use none of the tricks the “big boys” use (nmaps half-open SYN scans, etc.). But even for a simplistic scanner, this can be very useful.  I will be the first one to say I am no Python expert.  Some of the explanations may be inaccurate and undoubtedly there are better ways of doing this. Hopefully this demonstrates, that even with incomplete knowledge of  Python, you can rapidly turn your ideas into working code.

I look forward to seeing your comments, questions and corrections.  What awesome tools have you thrown together in Python, or your favorite programming language?

Hi Stumbleuponers

Recently I’ve gotten a burst of traffic from stumbleupon so I would like to thank you for checking the site out and taking the time to recommend it to others. If you enjoy what you’ve been reading, I would implore you to do two things:

Subscribe to the RSS feed

If you are using a spiffy browser it’s already up in the URL bar, just waiting for your click. It’s quick, it’s easy, all the cool kids are doing it.

Feedback

If you like the content let me know. If it sucks eggs, let me know that as well. One thing I’ve learned in my short time on this planet – conversations are more fun when they are read/write.

Thanks guys, stumble on.

Quickie: Finding Listening Ports

File this one with “things i constantly use but refuse to memorize so I always have to google it”…

The following command is a simple demonstration of the power of piping data from one output to another programs input.  Netstat will list all of the processes using network connects. Find will “filter” out only the ports in a Listening state.


netstat -an | find /i "listening"

Text based formats FTW.

Quick and Dirty Malware Removal

The following is a quick and dirty guide to virus/malware removal. These are simple, proven steps to clean out malware and get a PC back and running as quick as possible.  No attempt is made to do any sort of forensics nor are there any techniques included for measuring or controlling the propagation to other machines or networks.  The assumption is that the damage has already been done and you are just cleaning up the mess.

Arming Yourself

The following is a list of tools you will need to have handy.  It is highly recommended that you download these on uninfected machine before hand! There are many types of malware out there that are known to detect file names/ signatures of popular cleaning tools and infect them.  You have been warned!

Kill Everything That Moves

  • Alt-F4 any obvious pop-ups, don’t take the risk of click on the window even if the border “looks like a window”, it could easily be an imagemap
  • Launch process explorer and look through all of the running processes. Look very closely, do any of the names look out of place?  Eating up too much memory?  Pay special attention to the “Company Name”
  • Once you have killed any running processes, double check for open ports. Got to the command prompt and run netstat -an | find /i "listening" This will show you all ports that are being listened on.

Stop It From Spawning

  • Launch Autoruns and look through all of the start-up items.  There is a feature in the latest version to hide Microsoft signed entries, this will make your life much easier. Save the current settings in Autoruns before removing any items. That way if you remove something you shouldn’t you can replace it, Check all start-up items, also pay attention to drivers and service. Again there are many obvious telltale signs of malware – misspellings, invalid directories, etc.
  • Check Start->Program Files->Startup

Destroy the Source

Run Combofix. This can talk up 40 minutes to completely run.  For part it will need to be connected to the network to update.  During its run Combofix launches many small utilities that clean various malware. For part of its network cleaning/repair the machine will lose network connection for up to 10 mins.  If you are running Combofix remotely, don’t panic. Seriously, even when you think “God what have I done, I’ve killed the network connection!”, it will come back.

Clean Up the Mess

  • Run Malwarebytes quick scan
  • Visit Add/Remove programs and remove any “junk” programs.  Use Revouninstaller for stubborn ones

Sanity Check

  • Launch IE and verify functionality by going to a few websites.  Also do a search on Yahoo, Bing and Google.  Verify that the search result links actually take you to the correct site (not redirected).
  • Review c:\windows\system32\drivers\etc\hosts

Share your tips

Do you have any tips or tricks, any hand software that you use to defeat the ever growing scourge that is malware?

Jarlsberg Means Hacker Fun

This is just a quickie post, but I just heard about a new google Code Lab project called Jarlsberg.  Jarslberg is part tutorial and part toy web-app – it’s goal is to teach web developers common web vulnerabilities and how to defend against them.

I’m just digging in to this so I don’t have much to report yet, but it looks like a blast!

Have fun and happy hacking.

http://jarlsberg.appspot.com/

Phishing and Spam IQ Quiz – Will You Pass?

I just took the Sonicwall Phishing and Spam IQ Quiz and I’m proud to say I passed with flying colors.  All told, it probably took under 5 minutes to complete, while I was getting ready to leave for work. 

It’s doubtful if Sonicwallwill ever release any data from this quiz, but I would hazard a guess that the results wouldn’t surprise anyone.  Those who would describe themselves as “technical” probably got near to 100% in all cases, while those who could be described as non-technical probably scored similarly to random chance.  If you consider yourself techincal and still missed a few, consider this.  These emails are deliberately designed to be deceptive.  Any of us scoring 100% probably applied my more care and reasoning to the quiz then we normally do to our email.  If everyone went at their typical scanning speed, I’m sure there would have been no perfects!

So, why is it that to some of us these phishing scams are blindingly obvious?  I think the answer lies in what we look for.  For most of the technical audience taking the test, I would suspect they scanned the emails the same way I did.  Look only at the urls, if they “look” legitimate they probably are, if they “look” fake they are probably a scam.

My thought is that a less sophisticated user would probably read through the email, trying to weigh the tone of authority, the context and various other clues to determine if it was legitimate or not.  And therein lies the problem.  By even reading the email they have started to sell themselves into the “false context” trick of the social engineer.  People will do seemingly careless things (give away social security numbers, passwords, etc) if the context is crafted in a way to make them believe “this is okay”.

How can we defend against this, both for the people we may be educating , but also to ourselves?  My personal plan is to develop “shortcuts”.  Logical rules that will immediately tell me whether something is probably a scam or not.  That way I don’t allow myself to get caught up in the narrative and buy into the context offered.  For instance, in the quiz above, my personal “rule” or shortcut was to look at the url, if it looked odd I would assume that it was fake.  There is little to lose by doing this because if it turns out to be a legitimate request, someone will contact you another way.

Anyone out there brave enough to admit they have fallen for a scam?  What were the telltale signs, that are now obvious, that you didn’t pick up on?  Do you have any personal “shortcuts” that you use when evaluating emails, phonecalls or other requests for your personal information.

If you are a system administrator I implore you to create a similar test for your users and report back with the results!

Three lesser known security tools for your arsenal

Roadkil’s DHCP Find

A small and efficient, windows based, app to find rogue DHCP servers.  It works by simply sending out DHCP requests and logging all the servers that  reply.  Not much to it, but when you are tracking down DHCP servers, this will be extremely valuable. DHCP Find 1.2

Nast

Let’s face it, tools like tcpdump and nmap are unbelievably powerful,  but without a lot of patience and a lot of experience they can both be overwhelming. Enter “nast”.  Nast is like the swiss army knife you keep in your pocket.  Its a handy set of very useful tools for network troubleshooting (and mischief) all wrapped into one handy program.  It has increasingly become my “go to” tool when i just want to get a job done. Here is a sampling of its features:

  • Sniffing/Dumping packets in ascii, ascii hex, and tcpdump formats
  • Remote promiscuous mode checking – Who else is monitoring the network?
  • Host listing – build a quick list of available hosts using arp
  • Gateway discovery – Are there multiple ways out of your network?
  • Reset connection – Destroy a connection in progress.  This could be fun!
  • Port scanning –  A quick, half-open scan, noting possible firewall rules.  Again, this seems really speedy

See the nast homepage for source code, full man page and contact information

SSLStrip

Curious about what is actually being sent back and forth in your https session?  Take a little peak with sslstrip. Unlike our previous tools, sslstrip requires a little upfront work to get going. You will need a linux box to do the work on.

  • Turn on forwarding: echo "1" > /proc/sys/net/ipv4/ip_forward
  • Set iptables to redirect to whatever port you want sslstrip to listen on: iptables -t nat -A PREROUTING -p tcp --destination-port 80 -j REDIRECT --to-port
  • Start sslstrip sslstrip -w -l
  • In another screen tail -f output.log (if you want)
  • Now your box is ready to snoop on any forwarded ssl connections, lets send them our way. In another screen use the command arpspoof -i -t

With everything running, test out a few https webpages. You should see the contents of your http post appear in the logfile.  Check the options of sslstrip for more detailed logging and other features.

Sslstrip can be found at http://www.thoughtcrime.org/software/sslstrip/ .  Arpspoof is part of dnsiff, which can be found over here.

How dd, nc and fdisk saved my bacon

How do I get data off this thing?

Here’s the situation.  I’m trying to recover data for a client.  Their laptop will not boot into Windows and they need to get data off of it.  To add to the fun, I don’t have an IDE adaptor handy for the laptop harddrive and the laptop is a little crashy (motherboard shorting out?)

I could wait around until I can get my hands on an adaptor, but what fun would that be?

First things first, we need to get the data off of the drive.  Being a minimalist, I first thought of just doing a file copy.  Not knowing whether the file-system is any good, this probably isn’t a good idea.  In any forensics situation you always want to clone your source and then work with it.  i will be taking the same approach, treating the source as “read only” and only modifying the copies I make.  If it turns out the file-system is corrupt I can always clone to a known-good drive and use tools like “chkdsk” (for file-system corruption) or “partimage” (if things are really nasty).

So I have the “donor” machine booted up on a recent cd of Knoppix, connected via crossover cable to our tech station.  I have set ip addressed and tested connectivity between the machines.  Now it’s time for the magic to start.

Using the instructions found here I set my tech station as a netcat “server” and the dead laptop as a netcat “client”.  Score one for the simplicity of pipes, because dd will happily pipe bit for bit data across our netcat tunnel!

“Server”

nc -l -p 9000 | dd of=HarddriveImage.img

“Client”

dd if=/dev/sda | nc 192.168.1.220 9000

I also setup a terminal on the tech station running “watch” to keep an eye on the disk image size

watch -n 1 "du -hca *.img"

Now we wait…in my case 19 gigs…about one hour

We have the image, now what?

Next, I found a handy guide here on what to actually do with the image once I got it.  I am familiar with loopback mounting of iso images, but never before have I tried to loopback mount an entire hard drive image.

So first things first, we need one more important piece of information off the dying computer.  We need to know how many cylinders the hard drive has.  This will come into play, later, in our calculations.

fdisk -l /dev/sda

Disk /dev/sda: 160.0 GB, 160041885696 bytes
255 heads, 63 sectors/track, 19457 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00050229

Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          12       96358+  83  Linux
/dev/sda2              13         136      996030    5  Extended
/dev/sda3             137        1109     7815622+  83  Linux
/dev/sda4            1110       19457   147380310   83  Linux
/dev/sda5              13         136      995998+  82  Linux swap / Solaris

This is a different hard drive, so the cylinder count will vary from below.  The cylinder count for the disk we are using is 2432

Now that we know the cylinders we can try to get fdisk to read the image we cloned to see if it is intact

tech@tech-desktop:~/tmp$ fdisk -C 2432 DiskImage.img

The number of cylinders for this disk is set to 2432.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): p

Disk DiskImage.img: 0 MB, 0 bytes
255 heads, 63 sectors/track, 2432 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x9dc96e9e

Device Boot      Start         End      Blocks   Id  System
DiskImage.img1               1           5       40131   de  Dell Utility
DiskImage.img2   *           6        2431    19486845    7  HPFS/NTFS

Well, everything looks good!  If you have trouble at this point, keep in mind that the disk may be beyond saving.  Try something like Spinrite and attempt this again.

We know that we want to mount partition 2 “DiskImage.img2″ but we need to get the correct start and end blocks.

tech@tech-desktop:~/tmp$ fdisk -l -u -C 2432 DiskImage.img

Disk DiskImage.img: 0 MB, 0 bytes
255 heads, 63 sectors/track, 2432 cylinders, total 0 sectors
Units = sectors of 1 * 512 = 512 bytes
Disk identifier: 0x9dc96e9e

Device Boot      Start         End      Blocks   Id  System
DiskImage.img1              63       80324       40131   de  Dell Utility
DiskImage.img2   *       80325    39054014    19486845    7  HPFS/NTFS

Now all that is left is to calculate the offset so we can tell mount where to start mounting the loopback image.

Offset = StartNumber * 512

So we have 80325 * 512 = 41126400

No we issue the mount command as follows:

tech@tech-desktop:~/tmp$ sudo mount -o loop,offset=41126400 -t ntfs DiskImage.img /mnt/

That’s all she wrote

Well, we’ve been on a whirlwind tale of data recovery, but I am sure we have just scratched the surface of the different techniques that are out there.  So, what tricks have you used for data recovery before?  Any tips or tools you care to share?

Coming Soon…

It’s been a little few and far between lately, so I wanted to give everyone a taste of what is coming up.

Using Git as a “Poor Man’s” Time Machine – Part Two

Part two of our practical introduction of Git to those new to revision control

A Little Hacky System to Monitor Your Cron Jobs

We will get into a little Python / PHP action to create a very simple “dashboard” for your cronjobs

Erlang Weather Tracker

We are going to build a weather tracking node to nab data from NOAA as a gentle introduction to Erlang and functional programming

A Brief Description of a ZFS Based Backup System

I’m going to break down the different parts of the Helpful Hackers current backup system/scheme.  What’s good along with what can be improved