Using Operating System Wrappers to Increase the Resiliency of Commercial Firewalls

Using Operating System Wrappers to Increase the

Resiliency of Commercial Firewalls

Jeremy Epstein Linda Thomas Eric Monteith

jepstein@webMethods.com Ithomas@webMethods.com eric-monteith@nai.com

webMethods, Inc. webMethods, Inc. NAI Labs

Abstract'

Operating system wrappers technology provides a

means for providing fine grained controls on the operation

of applications software. Application proxy firewalls can

gain from this technology by wrapping the proxies, thus

preventing bugs (or malicious software) in the proxy from

subverting the intent of the firewall. This paper describes

several experiments we performed with wrappers and

firewalls, using several different firewalls and types of

wrappers.

1 Introduction

Access controls in operating systems are usually at a

coarse level and frequently do not cover all types of

resources in the system. For example, UNIX systems

control access to files, but the only controls on sockets

limit non-root processes from binding low numbered

sockets. Operating system wrapper technologies

(henceforth "wrappers"), including those described in

[Jones], [Fraser], [Balzer], among others, allow specifying

the behavior of application processes to an arbitrary level

ofgranuIarity.2

While wrapper technology is aimed at constraining the

behavior of applications on end systems (especially

clients, and possibly also servers), it is also applicable to

security devices such as firewalls. As part of the DARPA

Information Assurance program, we have performed a

series of experiments using different types of wrappers to

constrain the behavior of several different firewall

products. This paper describes the results of those

experiments, and points to directions for future research.

The remainder of this paper is organized as follows.

Section 2 describes our motivation for developing firewall

' The work described in this paper was performed while all three authors

were associated by NAI Labs. * The term "wrappers' is overloaded in the security field. In this paper,

it means fimctions that intercept system calls and perform mediation.

This is different from TCP Wrappers [Vmema] which are a program

between inefd and the service provider daemons, but do not attempt to

intercept system calls.

wrappers. While this paper assumes a basic

understanding of wrapper technology, Section 3 provides

a synopsis of what wrappers are and how they work, and

describes some of the differences between the wrappers

technology developed by NAI Labs EFraser] and the

wrappers technology developed by the Information

Sciences Institute (ISI) [Balzer]. Section 4 describes how

we wrapped the Gauntlet Internet Firewall (for which we

had design information and source code available) using

the NAI Labs wrappers. Section 5 describes our

experiences in using the NAI Labs wrappers to wrap

firewalls for which we had no source code or design

information. Section 6 describes how 1 we wrapped the

Gauntlet Intemet Firewall using the IS1 wrappers, and as

such is a parallel to Section 4. Section 7 gives our current

status and availability of our prototypes. Section 8

concludes the paper.

2 Motivation

Application level firewall proxies are fragile, and are

growing ever more complex. Customers demand

increasing functionality, including the ability to perform

tasks such as virus scanning, limits on addresses visited

(e.g., to prevent access to pornographic web sites), and

detailed scanning of protocols to prevent outsiders from

exploiting vulnerabilities in host systems. As the proxies

become increasingly complex, the likelihood of flaws that

allow security breaches increases. For example, it is

likely that there are opportunities in most firewall proxies

for buffer overrun attacks.

As the number of protocols increases, proxies are

increasingly written by people without hfficient training

in writing safe software. End users want to write their

own proxies, since they can do it more rapidly than

waiting for a firewall vendor to include a suitable proxy in

the product. While both vendors and end users make

reasonable