306 lines
20 KiB
HTML
306 lines
20 KiB
HTML
<html><head><meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"><title>Npcap internals</title><meta name="generator" content="DocBook XSL Stylesheets V1.79.2"><meta name="description" content="Describes the internal structure and interfaces of Npcap: the NPF driver and Packet.dll"><link rel="home" href="index.html" title="Npcap: Nmap Project's packet sniffing library for Windows"><link rel="up" href="index.html" title="Npcap: Nmap Project's packet sniffing library for Windows"><link rel="prev" href="npcap-tutorial.html" title="Npcap Development Tutorial"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">Npcap internals</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="npcap-tutorial.html">Prev</a> </td><th width="60%" align="center"> </th><td width="20%" align="right"> </td></tr></table><hr></div><div class="sect1"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="npcap-internals"></a>Npcap internals</h2></div><div><div class="abstract"><p class="title"><b>Abstract</b></p>
|
||
<p>Describes the internal structure and interfaces of Npcap: the NPF
|
||
driver and Packet.dll</p>
|
||
</div></div></div></div>
|
||
|
||
|
||
<p>This portion of the manual describes the internal structure and
|
||
interfaces of Npcap, starting from the lowest-level module. It is targeted
|
||
at people who must extend or modify this software, or to the ones
|
||
interested in how it works. Therefore, developers who just want to use
|
||
Npcap in their software don't need to read it.</p>
|
||
|
||
<div class="sect2"><div class="titlepage"><div><div><h3 class="title"><a name="npcap-structure"></a>Npcap structure</h3></div></div></div>
|
||
|
||
|
||
<p>Npcap is an architecture for packet capture and network analysis for the
|
||
Win32 platforms. It includes a kernel-level packet filter, a
|
||
low-level dynamic link library (packet.dll), and a high-level and
|
||
system-independent library (wpcap.dll).</p>
|
||
|
||
<p>Why do we use the term <em class="wordasword">architecture</em> rather
|
||
than <em class="wordasword">library</em>? Because packet capture is a low
|
||
level mechanism that requires a strict interaction with the network
|
||
adapter and with the operating system, in particular with its networking
|
||
implementation, so a simple library is not sufficient.</p>
|
||
|
||
|
||
<div class="sect3"><div class="titlepage"><div><div><h4 class="title"><a name="id572152"></a>Main components of Npcap.</h4></div></div></div>
|
||
|
||
|
||
<p>First, a capture system needs to bypass the operating systems's
|
||
protocol stack in order to access the raw data transiting on the
|
||
network. This requires a portion running inside the kernel of OS,
|
||
interacting directly with the network interface drivers. This portion
|
||
is very system dependent, and in our solution it is realized as a
|
||
device driver, called Netgroup Packet Filter (NPF); This driver offers
|
||
basic features like packet capture and injection, as well as more
|
||
advanced ones like a programmable filtering system and a monitoring
|
||
engine. The filtering system can be used to restrict a capture session
|
||
to a subset of the network traffic (e.g. it is possible to capture only
|
||
the ftp traffic generated by a particular host); the monitoring engine
|
||
provides a powerful but simple to use mechanism to obtain statistics on
|
||
the traffic (e.g. it is possible to obtain the network load or the
|
||
amount of data exchanged between two hosts).</p>
|
||
|
||
<p>Second, the capture system must export an interface that user-level
|
||
applications will use to take advantage of the features provided by the
|
||
kernel driver. Npcap provides two different libraries:
|
||
<code class="filename">packet.dll</code> and
|
||
<code class="filename">wpcap.dll</code>.</p>
|
||
|
||
<p> Packet.dll offers a low-level API that can be used to directly
|
||
access the functions of the driver, with a programming interface
|
||
independent from the Microsoft OS.</p>
|
||
|
||
|
||
<p>Wpcap.dll exports a more powerful set of high level capture
|
||
primitives that are compatible with libpcap, the well known Unix
|
||
capture library. These functions enable packet capture in a manner that
|
||
is independent of the underlying network hardware and operating
|
||
system.</p>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="sect2"><div class="titlepage"><div><div><h3 class="title"><a name="npcap-internals-driver"></a>Npcap driver internals</h3></div></div></div>
|
||
|
||
|
||
<p>This section documents the internals of the Netgroup Packet Filter
|
||
(NPF), the kernel portion of Npcap. Normal users are probably interested
|
||
in how to use Npcap and not in its internal structure. Therefore the
|
||
information present in this module is destined mainly to Npcap developers
|
||
and maintainers, or to the people interested in how the driver works. In
|
||
particular, a good knowledge of OSes, networking and Windows kernel
|
||
programming and device drivers development is required to profitably read
|
||
this section.</p>
|
||
|
||
<p>NPF is the Npcap component that does the hard work, processing the
|
||
packets that transit on the network and exporting capture, injection and
|
||
analysis capabilities to user-level.</p>
|
||
|
||
<p>The following paragraphs will describe the interaction of NPF with
|
||
the OS and its basic structure.</p>
|
||
|
||
<div class="sect3"><div class="titlepage"><div><div><h4 class="title"><a name="npcap-internals-driver-ndis"></a>NPF and NDIS</h4></div></div></div>
|
||
|
||
|
||
<p>NDIS (Network Driver Interface Specification) is a standard that
|
||
defines the communication between a network adapter (or, better, the
|
||
driver that manages it) and the protocol drivers (that implement for
|
||
example TCP/IP). Main NDIS purpose is to act as a wrapper that allows
|
||
protocol drivers to send and receive packets onto a network (LAN or
|
||
WAN) without caring either the particular adapter or the particular
|
||
Win32 operating system.</p>
|
||
|
||
<p>NDIS supports four types of network drivers:</p>
|
||
|
||
<div class="orderedlist"><ol class="orderedlist" type="1"><li class="listitem">
|
||
<p><span class="emphasis"><em>Miniport drivers</em></span>. Miniport drivers
|
||
directly manage network interface cards, referred to as NICs. The
|
||
miniport drivers interface directly to the hardware at their lower
|
||
edge and at their upper edge present an interface to allow upper
|
||
layers to send packets on the network, to handle interrupts, to
|
||
reset the NIC, to halt the NIC and to query and set the operational
|
||
characteristics of the driver.</p>
|
||
<p>Miniport drivers implement only the hardware-specific
|
||
operations necessary to manage a NIC, including sending and
|
||
receiving data on the NIC. Operations common to all lowest level
|
||
NIC drivers, such as synchronization, is provided by NDIS.
|
||
Miniports do not call operating system routines directly; their
|
||
interface to the operating system is NDIS.</p>
|
||
<p>A miniport does not keep track of bindings. It merely passes
|
||
packets up to NDIS and NDIS makes sure that these packets are
|
||
passed to the correct protocols.</p>
|
||
</li><li class="listitem">
|
||
<p><span class="emphasis"><em>Intermediate drivers</em></span>. Intermediate drivers
|
||
interface between an upper-level driver such as a protocol driver
|
||
and a miniport. To the upper-level driver, an intermediate driver
|
||
looks like a miniport. To a miniport, the intermediate driver looks
|
||
like a protocol driver. An intermediate protocol driver can layer
|
||
on top of another intermediate driver although such layering could
|
||
have a negative effect on system performance. A typical reason for
|
||
developing an intermediate driver is to perform media translation
|
||
between an existing legacy protocol driver and a miniport that
|
||
manages a NIC for a new media type unknown to the protocol driver.
|
||
For instance, an intermediate driver could translate from LAN
|
||
protocol to ATM protocol. An intermediate driver cannot communicate
|
||
with user-mode applications, but only with other NDIS drivers.</p>
|
||
</li><li class="listitem">
|
||
<p><span class="emphasis"><em>Filter drivers</em></span>. Filter drivers can monitor
|
||
and modify traffic between protocol drivers and miniport drivers
|
||
like an intermediate driver, but are much simpler. They have less
|
||
processing overhead than intermediate drivers.</p>
|
||
</li><li class="listitem">
|
||
<p><span class="emphasis"><em>Transport drivers or protocol drivers</em></span>. A
|
||
protocol driver implements a network protocol stack such as IPX/SPX
|
||
or TCP/IP, offering its services over one or more network interface
|
||
cards. A protocol driver services application-layer clients at its
|
||
upper edge and connects to one or more NIC driver(s) or
|
||
intermediate NDIS driver(s) at its lower edge.</p>
|
||
</li></ol></div>
|
||
|
||
<p>NPF is implemented as a filter driver. In order to provide complete
|
||
access to the raw traffic and allow injection of packets, it is
|
||
registered as a modifying filter driver in the compression
|
||
<code class="literal">FilterClass</code>.</p>
|
||
|
||
<p>Notice that the various Windows operating systems have different
|
||
versions of NDIS: NPF is NDIS 6.0 compliant, and so requires a Windows
|
||
OS that supports NDIS 6.0: Windows Vista or later.</p>
|
||
|
||
|
||
|
||
</div>
|
||
<div class="sect3"><div class="titlepage"><div><div><h4 class="title"><a name="npcap-internals-structure"></a>NPF structure basics</h4></div></div></div>
|
||
|
||
|
||
|
||
|
||
<p>NPF is able to perform a number of different operations: capture,
|
||
monitoring, dump to disk, packet injection. The following paragraphs
|
||
will describe shortly each of these operations.</p>
|
||
<div class="sect4"><div class="titlepage"><div><div><h5 class="title"><a name="npcap-internals-capture"></a>Packet Capture</h5></div></div></div>
|
||
|
||
|
||
<p>The most important operation of NPF is packet capture. During a
|
||
capture, the driver sniffs the packets using a network interface and
|
||
delivers them intact to the user-level applications.</p>
|
||
<p>The capture process relies on two main components:</p>
|
||
<div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem"><p>A packet filter that decides if an incoming packet
|
||
has to be accepted and copied to the listening application. Most
|
||
applications using NPF reject far more packets than those
|
||
accepted, therefore a versatile and efficient packet filter is
|
||
critical for good over-all performance. A packet filter is a
|
||
function with boolean output that is applied to a packet. If the
|
||
value of the function is true the capture driver copies the
|
||
packet to the application; if it is false the packet is
|
||
discarded. NPF packet filter is a bit more complex, because it
|
||
determines not only if the packet should be kept, but also the
|
||
amount of bytes to keep. The filtering system adopted by NPF
|
||
derives from the <span class="emphasis"><em>BSD Packet Filter</em></span> (BPF), a
|
||
virtual processor able to execute filtering programs expressed in
|
||
a pseudo-assembler and created at user level. The application
|
||
takes a user-defined filter (e.g. <span class="quote">“<span class="quote">pick up all UDP
|
||
packets</span>”</span>) and, using wpcap.dll, compiles them into a BPF
|
||
program (e.g. <span class="quote">“<span class="quote">if the packet is IP and the
|
||
<code class="literal">protocol type</code> field is equal to 17, then
|
||
return true</span>”</span>). Then, the application uses the
|
||
<code class="literal">BIOCSETF</code> IOCTL to inject the filter in the
|
||
kernel. At this point, the program is executed for every incoming
|
||
packet, and only the conformant packets are accepted. Unlike
|
||
traditional solutions, NPF does not
|
||
<span class="emphasis"><em>interpret</em></span> the filters, but it
|
||
<span class="emphasis"><em>executes</em></span> them. For performance reasons,
|
||
before using the filter NPF feeds it to a JIT compiler that
|
||
translates it into a native 80x86 function. When a packet is
|
||
captured, NPF calls this native function instead of invoking the
|
||
filter interpreter, and this makes the process very fast. The
|
||
concept behind this optimization is very similar to the one of
|
||
Java jitters.</p>
|
||
</li><li class="listitem">
|
||
<p>A circular buffer to store the packets and avoid loss. A
|
||
packet is stored in the buffer with a header that maintains
|
||
information like the timestamp and the size of the packet.
|
||
Moreover, an alignment padding is inserted between the packets in
|
||
order to speed-up the access to their data by the applications.
|
||
Groups of packets can be copied with a single operation from the
|
||
NPF buffer to the applications. This improves performances
|
||
because it minimizes the number of reads. If the buffer is full
|
||
when a new packet arrives, the packet is discarded and hence it's
|
||
lost. Both kernel and user buffer can be changed at runtime for
|
||
maximum versatility: packet.dll and wpcap.dll provide functions
|
||
for this purpose.</p>
|
||
</li></ul></div>
|
||
|
||
<p>The size of the user buffer is very important because it determines
|
||
the <span class="emphasis"><em>maximum</em></span> amount of data that can be copied from
|
||
kernel space to user space within a single system call. On the other
|
||
hand, it can be noticed that also the <span class="emphasis"><em>minimum</em></span>
|
||
amount of data that can be copied in a single call is extremely
|
||
important. In presence of a large value for this variable, the kernel
|
||
waits for the arrival of several packets before copying the data to the
|
||
user. This guarantees a low number of system calls, i.e. low processor
|
||
usage, which is a good setting for applications like sniffers. On the
|
||
other side, a small value means that the kernel will copy the packets
|
||
as soon as the application is ready to receive them. This is excellent
|
||
for real time applications (like, for example, ARP redirectors or
|
||
bridges) that need the better responsiveness from the kernel. From
|
||
this point of view, NPF has a configurable behavior, that allows users
|
||
to choose between best efficiency or best responsiveness (or any
|
||
intermediate situation).</p>
|
||
|
||
<p>The wpcap library includes a couple of system calls that can be
|
||
used both to set the timeout after which a read expires and the minimum
|
||
amount of data that can be transferred to the application. By default,
|
||
the read timeout is 1 second, and the minimum amount of data copied
|
||
between the kernel and the application is 16K.</p>
|
||
</div>
|
||
<div class="sect4"><div class="titlepage"><div><div><h5 class="title"><a name="npcap-internals-injection"></a>Packet injection</h5></div></div></div>
|
||
|
||
|
||
<p>NPF allows to write raw packets to the network. To send data, a
|
||
user-level application performs a WriteFile() system call on the NPF
|
||
device file. The data is sent to the network as is, without
|
||
encapsulating it in any protocol, therefore the application will have
|
||
to build the various headers for each packet. The application usually
|
||
does not need to generate the FCS because it is calculated by the
|
||
network adapter hardware and it is attached automatically at the end of
|
||
a packet before sending it to the network.</p>
|
||
|
||
<p>In normal situations, the sending rate of the packets to the
|
||
network is not very high because of the need of a system call for each
|
||
packet. For this reason, the possibility to send a single packet more
|
||
than once with a single write system call has been added. The
|
||
user-level application can set, with an IOCTL call
|
||
(<code class="literal">BIOCSWRITEREP</code>), the number of times a single packet
|
||
will be repeated: for example, if this value is set to 1000, every raw
|
||
packet written by the application on the driver's device file will be
|
||
sent 1000 times. This feature can be used to generate high speed
|
||
traffic for testing purposes: the overload of context switches is no
|
||
longer present, so performance is remarkably better.</p>
|
||
</div>
|
||
|
||
<div class="sect4"><div class="titlepage"><div><div><h5 class="title"><a name="npcap-internals-monitoring"></a>Network monitoring</h5></div></div></div>
|
||
|
||
|
||
<p>Npcap offers a kernel-level programmable monitoring module, able to
|
||
calculate simple statistics on the network traffic. Statistics can be
|
||
gathered without the need to copy the packets to the application, that
|
||
simply receives and displays the results obtained from the monitoring
|
||
engine. This allows to avoid great part of the capture overhead in
|
||
terms of memory and CPU clocks.</p>
|
||
|
||
<p>The monitoring engine is made of a <span class="emphasis"><em>classifier</em></span>
|
||
followed by a <span class="emphasis"><em>counter</em></span>. The packets are classified
|
||
using the filtering engine of NPF, that provides a configurable way to
|
||
select a subset of the traffic. The data that pass the filter go to the
|
||
counter, that keeps some variables like the number of packets and the
|
||
amount of bytes accepted by the filter and updates them with the data
|
||
of the incoming packets. These variables are passed to the user-level
|
||
application at regular intervals whose period can be configured by the
|
||
user. No buffers are allocated at kernel and user level.</p>
|
||
</div>
|
||
|
||
|
||
</div>
|
||
</div>
|
||
<div class="sect2"><div class="titlepage"><div><div><h3 class="title"><a name="npcap-internals-references"></a>Further reading</h3></div></div></div>
|
||
|
||
<p>The structure of NPF and its filtering engine derive directly from
|
||
the one of the BSD Packet Filter (BPF), so if you are interested the
|
||
subject you can read the following papers:</p>
|
||
<div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem"><p>S. McCanne and V. Jacobson, <a class="ulink" href="ftp://ftp.ee.lbl.gov/papers/bpf-usenix93.ps.Z" target="_top">The BSD Packet
|
||
Filter: A New Architecture for User-level Packet Capture</a>.
|
||
Proceedings of the 1993 Winter USENIX Technical Conference (San
|
||
Diego, CA, Jan. 1993), USENIX.</p>
|
||
</li><li class="listitem"><p>A. Begel, S. McCanne, S.L.Graham, BPF+: <a class="ulink" href="http://www.acm.org/pubs/articles/proceedings/comm/316188/p123-begel/p123-begel.pdf" target="_top">Exploiting
|
||
Global Data-flow Optimization in a Generalized Packet Filter
|
||
Architecture</a>, Proceedings of ACM SIGCOMM '99, pages 123-134,
|
||
Conference on Applications, technologies, architectures, and
|
||
protocols for computer communications, August 30 - September 3, 1999,
|
||
Cambridge, USA</p>
|
||
</li></ul></div>
|
||
</div>
|
||
|
||
</div><div class="navfooter"><hr><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="npcap-tutorial.html">Prev</a> </td><td width="20%" align="center"> </td><td width="40%" align="right"> </td></tr><tr><td width="40%" align="left" valign="top">Npcap Development Tutorial </td><td width="20%" align="center"><a accesskey="h" href="index.html">Home</a></td><td width="40%" align="right" valign="top"> </td></tr></table></div></body></html>
|