Starting commit for documentation & genetic algorithm

pull/17/head
Kkevsterrr 4 years ago
parent 502900e3e5
commit 5f13b926e3

@ -9,8 +9,12 @@ exclude_lines =
raise NotImplementedError
if __name__ == .__main__.:
def get_args
def main
except KeyboardInterrupt:
ignore_errors = True
omit =
experimental/*
experimental/smtp/*
tests/*
examples/*
censors/*
censor_driver.py
graph.py

@ -1,39 +1,32 @@
sudo: required
dist: "bionic"
services:
- docker
language: python
python:
- "3.6"
install:
# Travis recently added systemd-resolvd to their VMs. Since full Geneva often runs its own DNS
# server to test DNS strategies, we need to disable system-resolvd.
# First disable the service
- sudo systemctl disable systemd-resolved.service
# Stop the service
- sudo systemctl stop systemd-resolved
# With systemd not running, our own hostname won't resolve - this causes issues with sudo.
# Add back our hostname to /etc/hosts/ so sudo does not complain
- echo $(hostname -I | cut -d\ -f1) $(hostname) | sudo tee -a /etc/hosts
# Replace the 127.0.0.53 nameserver with Google's
- sudo sed 's/nameserver.*/nameserver 8.8.8.8/' /etc/resolv.conf > /tmp/resolv.conf.new
# Travis recently added systemd-resolvd to their VMs. We must disable this to test our own DNS server
- sudo systemctl disable systemd-resolved.service # First disable the service
- sudo systemctl stop systemd-resolved # Stop the service
- echo $(hostname -I | cut -d\ -f1) $(hostname) | sudo tee -a /etc/hosts # add back our hostname to /etc/hosts so sudo does not complain
- sudo sed 's/nameserver.*/nameserver 8.8.8.8/' /etc/resolv.conf > /tmp/resolv.conf.new # replace the 127.0.0.53 nameserver with Google's
- sudo mv /tmp/resolv.conf.new /etc/resolv.conf
# Now that systemd-resolv.conf is safely disabled, we can now setup for Geneva
- sudo systemctl start docker # must now restart docker so the resolv.conf changes propagate to its containers
- sudo apt-get clean # travis having mirror sync issues
# Install dependencies
- sudo apt-get update
- sudo apt-get -y install libnetfilter-queue-dev python3 python3-pip python3-setuptools graphviz
# Since sudo is required but travis does not set up the root environment, we must override the
# secure_path in sudoers in order for travis's setup to take effect for sudo commands
- printf "Defaults\tenv_reset\nDefaults\tmail_badpass\nDefaults\tsecure_path="/home/travis/virtualenv/python3.6.7/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin"\nroot\tALL=(ALL:ALL) ALL\n#includedir /etc/sudoers.d\n" > /tmp/sudoers.tmp
# Verify the sudoers file
- sudo visudo -c -f /tmp/sudoers.tmp
# Copy in the sudoers file
- sudo cp /tmp/sudoers.tmp /etc/sudoers
# Now that sudo is good to go, finish installing dependencies
- printf "Defaults\tenv_reset\nDefaults\tmail_badpass\nDefaults\tsecure_path="/home/travis/virtualenv/python3.6.7/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin"\nroot\tALL=(ALL:ALL) ALL\n#includedir /etc/sudoers.d\n" > /tmp/sudoers.tmp # Since sudo is required but travis does not set up the root environment, we must override the secure_path in sudoers in order for travis's setup to take effect for sudo commands
- sudo visudo -c -f /tmp/sudoers.tmp # Verify the sudoers file
- sudo cp /tmp/sudoers.tmp /etc/sudoers # Copy in the sudoers file
- sudo echo $PATH # Confirm the PATH changes took effect
- sudo python3 -m pip install -r requirements.txt
- sudo python3 -m pip install slackclient pytest-cov
- docker build -t base:latest -f docker/Dockerfile .
script:
- sudo python3 -m pytest --cov=./ -sv tests/ --tb=short

@ -1,15 +1,17 @@
# Geneva [![Build Status](https://travis-ci.com/Kkevsterrr/geneva.svg?branch=master)](https://travis-ci.com/Kkevsterrr/geneva) [![codecov](https://codecov.io/gh/Kkevsterrr/geneva/branch/master/graph/badge.svg)](https://codecov.io/gh/Kkevsterrr/geneva)
Geneva is an artificial intelligence tool that defeats censorship by exploiting bugs in censors, such as those in China, India, and Kazakhstan. Unlike many other anti-censorship solutions which require assistance from outside the censoring regime (Tor, VPNs, etc.), Geneva runs strictly on the client.
Geneva is an artificial intelligence tool that defeats censorship by exploiting bugs in censors, such as those in China, India, and Kazakhstan. Unlike many other anti-censorship solutions which require assistance from outside the censoring regime (Tor, VPNs, etc.), Geneva runs strictly on one side of the connection (either the client or server side).
Under the hood, Geneva uses a genetic algorithm to evolve censorship evasion strategies and has found several previously unknown bugs in censors. Geneva's strategies manipulate the client's packets to confuse the censor without impacting the client/server communication. This makes Geneva effective against many types of in-network censorship (though it cannot be used against IP-blocking censorship).
Under the hood, Geneva uses a genetic algorithm to evolve censorship evasion strategies and has found several previously unknown bugs in censors. Geneva's strategies manipulate the network stream to confuse the censor without impacting the client/server communication. This makes Geneva effective against many types of in-network censorship (though it cannot be used against IP-blocking censorship).
This code release specifically contains the strategy engine used by Geneva, its Python API, and a subset of published strategies, so users and researchers can test and deploy Geneva's strategies. To learn more about how Geneva works, visit [How it Works](#How-it-Works). We will be releasing the genetic algorithm at a later date.
Geneva is composed of two high level components: its genetic algorithm (which it uses to evolve new censorship evasion strategies) and its strategy engine (which is uses to run an individual censorship evasion strategy over a network connection).
This codebase contains the Geneva's full implementation: its genetic algorithm, strategy engine, Python API, and a subset of published strategies. With these tools, users and researchers alike can evolve new strategies or leverage existing strategies to evade censorship. To learn more about how Geneva works, see [How it Works](#How-it-Works).
## Setup
Geneva has been developed and tested for Centos or Debian-based systems. Due to limitations of
netfilter and raw sockets, Geneva does not work on OS X or Windows at this time and requires *python3.6* (with more versions coming soon).
netfilter and raw sockets, Geneva does not work on OS X or Windows at this time and requires *python3.6*.
Install netfilterqueue dependencies:
```
@ -21,7 +23,10 @@ Install Python dependencies:
# python3 -m pip install -r requirements.txt
```
## Running it
## Running a Strategy
A censorship evasion strategy is simply a _description of how network traffic should be modified_. A strategy is not
code, it is a description that tells the engine how it should operate over traffic. For a fuller description of the DNA syntax, see [Censorship Evasion Strategies](#Censorship-Evasion-Strategies).
```
# python3 engine.py --server-port 80 --strategy "[TCP:flags:PA]-duplicate(tamper{TCP:dataofs:replace:10}(tamper{TCP:chksum:corrupt},),)-|" --log debug
@ -38,27 +43,67 @@ this will fail. To fix this, remove those rules.
## Strategy Library
Geneva has found dozens of strategies that work against censors in China, Kazakhstan, and India. We include several of these strategies in [strategies.md](strategies.md). Note that this file contains success rates for each individual country; a strategy that works in one country may not work as well as other countries.
Geneva has found dozens of strategies that work against censors in China, Kazakhstan, India, and Iran. We include several of these strategies in [strategies.md](strategies.md). Note that this file contains success rates for each individual country; a strategy that works in one country may not work as well as other countries.
Researchers have observed that strategies may have differing success rates based on your exact location. Although we have not observed this from our vantage points, you may find that some strategies may work differently in a country we have tested. If this is the case, don't be alarmed. However, please feel free to reach out to a member of the team directly or open an issue on this page so we can track how the strategies work from other geographic locations.
## Disclaimer
Running these strategies may place you at risk if you use it within a censoring regime. Geneva takes overt actions that interfere with the normal operations of a censor and its strategies are detectable on the network. Geneva is not an anonymity tool, nor does it encrypt any traffic. Understand the risks of running Geneva in your country before trying it.
Running these strategies may place you at risk if you use it within a censoring regime. Geneva takes overt actions that interfere with the normal operations of a censor and its strategies are detectable on the network. During the training process, Geneva will intentionally trip censorship many times. Geneva is not an anonymity tool, nor does it encrypt any traffic. Understand the risks of running Geneva in your country before trying it.
-------
## How it Works
# How it Works
See our [paper](#Paper) for an in-depth read on how Geneva works. Below is a walkthrough of the main concepts behind Geneva, the major components of the codebase, and how they can be used.
## Censorship Evasion Strategies
A censorship evasion strategy is simply a _description of how network traffic should be modified_. A strategy is _not
code_, it is a description that tells Geneva's stratgy engine how it should manipulate network traffic. The goal of a censorship evasion strategy is to modify the network traffic in a such a way that the censor is unable to censor it, but the client/server communication is unimpacted.
A censorship evasion strategy composed of one or more packet-level building blocks. Geneva's core building blocks are:
1. `duplicate`: takes one packet and returns two copies of the packet
2. `drop`: takes one packet and returns no packets (drops the packet)
3. `tamper`: takes one packet and returns the modified packet
4. `fragment`: takes one packet and returns two fragments or two segments
Since `duplicate` and `fragment` introduce _branching_, these actions are composed into a binary-tree structure called an _action tree_. Each tree also has a _trigger_. The trigger describes which packets the tree should run on, and the tree describes what should happen to each of those packets when the trigger fires. Once a trigger fires on a packet, it pulls the packet into the tree for modifications, and the packets that emerge from the tree are sent on the wire. Recall that Geneva operates at the packet level, therefore all triggers are packet-level triggers.
Multiple action trees together form a _forest_. Geneva handles outbound and inbound packets differently, so strategies are composed of two forests: an outbound forest and an inbound forest.
Consider the following example of a simple Geneva strategy.
```
+---------------+
| TCP:flags:A | <-- triggers on TCP packets with the flags field set to 'ACK'
+-------+-------+ matching packets are captured and pulled into the tree
|
+---------v---------+
duplicate <-- makes two copies of the given packet. the tree is processed
+---------+---------+ with an inorder traversal, so the left side is run first
|
+-------------+------------+
| |
+------------v----------+ v <-- dupilcate has no right child, so this packet will be sent on the wire unimpacted
tamper
{TCP:flags:replace:R} <-- parameters to this action describe how the packet should be tampered
+------------+----------+
|
+------------v----------+
tamper
{TCP:chksum:corrupt}
+------------+----------+
|
v <-- packets that emerge from an in-order traversal of the leaves are sent on the wire
See our paper for an in-depth read on how Geneva works. Below is a rundown of the format of Geneva's strategy DNA.
```
### Strategy DNA
This strategy triggers on `TCP` packets with the `flags` field set to `ACK`. It makes a duplicate of the `ACK` packet; the first duplicate has its flags field changed to `RST` and its checksum (`chksum`) field corrupted; the second duplicate is unchaged. Both packets are then sent on the network.
Geneva's strategies can be arbitrarily complicated, and it defines a well-formatted syntax for
expressing strategies to the engine.
### Strategy DNA
A strategy is simply a _description of how network traffic should be modified_. A strategy is not
code, it is a description that tells the engine how it should operate over traffic.
These strategies can be arbitrarily complicated, and Geneva defines a well-formatted string syntax for
unambiguously expressing strategies.
A strategy divides how it handles outbound and inbound packets: these are separated in the DNA by a
"\\/". Specifically, the strategy format is `<outbound forest> \/ <inbound forest>`. If `\/` is not
@ -66,35 +111,50 @@ present in a strategy, all of the action trees are in the outbound forest.
Both forests are composed of action trees, and each forest is allowed an arbitrarily many trees.
An action tree is comprised of a _trigger_ and a _tree_. The trigger describes _when_ the strategy
should run, and the tree describes what should happen when the trigger fires. Recall that Geneva
operates at the packet level, therefore all triggers are packet-level triggers. Action trees start
with a trigger, and always end with a `-|`.
Action trees always start with a trigger, which is formatted as: `[<protocol>:<field>:<value>]`. For example, the trigger: `[TCP:flags:S]` will run its corresponding tree whenever it sees a `TCP` packet with the `flags` field set to `SYN`. If the corresponding action tree is `[TCP:flags:S]-drop-|`, this action tree will cause the engine to drop any `SYN` packets. `[TCP:flags:S]-duplicate-|` will cause the engine to duplicate any SYN packets. Triggers also can contain an optional 4th parameter for _gas_, which describes the number of times a trigger can fire. The triger `[IP:version:4:4]` will run only on the first 4 IPv4 packets it sees. If the gas is negative, the trigger acts as a _bomb_ trigger, which means the trigger will not fire until a certain number of applicable packets have been seen. For example, the trigger `[IP:version:4:-2]` will trigger only after it has seen two matching packets (and it will not trigger on those first packets).
Triggers operate as exact-matches, are formatted as follows: `[<protocol>:<field>:<value>]`. For
example, the trigger: `[TCP:flags:S]` will run its corresponding tree whenever it sees a `SYN`
TCP packet. If the corresponding action tree is `[TCP:flags:S]-drop-|`, this action tree will cause
the engine to drop any `SYN` packets. `[TCP:flags:S]-duplicate-|` will cause the engine to
duplicate the SYN packet.
Syntactically, action trees end with `-|`.
Depending on the type of action, some actions can have up to two children. These are represented
Depending on the type of action, some actions can have up to two children (such as `duplicate`). These are represented
with the following syntax: `[TCP:flags:S]-duplicate(<left_child>,<right_child>)-|`, where
`<left_child>` and `<right_child>` themselves are trees. If `(,)` is not specified, any packets
that emerge from the action will be sent on the wire.
that emerge from the action will be sent on the wire. If an action only has one child (such as `tamper`), it is always the left child. `[TCP:flags:S]-tamper{<parameters>}(<left_child>,)-|`
Actions that have parameters specify those parameters within `{}`. For example, giving parameters to the `tamper` action could look like: `[TCP:flags:S]-tamper{TCP:flags:replace:A}-|`. This strategy would trigger on TCP `SYN` packets and replace the TCP `flags` field to `ACK`.
Any action that has parameters associated with it contain those parameters in `{}`. Consider the
following strategy with `tamper`.
Putting this all together, below is the strategy DNA representation of the above diagram:
```
[TCP:flags:A]-duplicate(tamper{TCP:flags:replace:R},)-| \/
[TCP:flags:A]-duplicate(tamper{TCP:flags:replace:R}(tamper{TCP:chksum:corrupt},),)-| \/
```
This strategy takes outbound `ACK` packets and duplicates them. To the first duplicate, it tampers
the packet by replacing the `TCP` `flags` field with `RST`, and does nothing to the second
duplicate.
Note that due to NFQueue limitations, actions that introduce branching (fragment, duplicate) are
Geneva has code to parse this strategy DNA into strategies that can be applied to network traffic using the engine.
Note that due to limitations of Scapy and NFQueue, actions that introduce branching (`fragment`, `duplicate`) are
disabled for incoming action forests.
-------
## Engine
The strategy engine (`engine.py`) applies a strategy to a network connection. The engine works by capturing all traffic to/from a specified port. Packets that match an active trigger are run through the associated action-tree, and packets that emerge from the tree are sent on the wire.
The engine also has a Python API for using it in your application. It can be used as a context manager or invoked in the background as a thread.
For example, consider the following simple application.
```python
import os
import engine
# Port to run the engine on
port = 80
# Strategy to use
strategy = "[TCP:flags:A]-duplicate(tamper{TCP:flags:replace:R}(tamper{TCP:chksum:corrupt},),)-| \/"
# Create the engine in debug mode
with engine.Engine(port, strategy, log_level="debug") as eng:
os.system("curl http://example.com?q=ultrasurf")
```
This script creates an instance of the engine with a specified strategy, and that strategy will be running for everything within the context manager. When the context manager exits, the engine will clean itself up. See the `examples/` folder for more use cases of the engine.
Due to limitations of scapy and NFQueue, the engine cannot be used to communicate with localhost.
## Citation

@ -1,7 +1,5 @@
"""
Action
Geneva object for defining a packet-level action.
Geneva superclass object for defining a packet-level action.
"""
import inspect
@ -17,16 +15,24 @@ ACTION_CACHE["in"] = {}
ACTION_CACHE["out"] = {}
BASEPATH = os.path.sep.join(os.path.dirname(os.path.abspath(__file__)).split(os.path.sep)[:-1])
class Action():
"""
Defines the superclass for a Geneva Action.
"""
# Give each Action a unique ID - this is needed for graphing/visualization
ident = 0
# Each Action has a 'frequency' field - this defines how likely it is to be chosen
# when a new action is chosen
frequency = 0
def __init__(self, action_name, direction):
"""
Initializes this action object.
Args:
action_name (str): Name of this action ("duplicate")
direction (str): Direction of this action ("out", "both", "in")
"""
self.enabled = True
self.action_name = action_name
@ -45,6 +51,12 @@ class Action():
"""
Returns whether this action applies to the given direction, as
branching actions are not supported on inbound trees.
Args:
direction (str): Direction to check if this action applies ("out", "in", "both")
Returns:
bool: whether or not this action can be used to a given direction
"""
if direction == self.direction or self.direction == "both":
return True
@ -67,6 +79,14 @@ class Action():
Dynamically imports all of the Action classes in this directory.
Will only return terminal actions if terminal is set to True.
Args:
direction (str): Limit imported actions to just those that can run to this direction ("out", "in", "both")
disabled (list, optional): list of actions that are disabled
allow_terminal (bool): whether or not terminal actions ("drop") should be imported
Returns:
dict: Dictionary of imported actions
"""
if disabled is None:
disabled = []
@ -84,7 +104,6 @@ class Action():
else:
return ACTION_CACHE[direction][terminal]
collected_actions = []
# Get the base path for the project relative to this file
path = os.path.join(BASEPATH, "actions")
@ -117,6 +136,14 @@ class Action():
def parse_action(str_action, direction, logger):
"""
Parses a string action into the action object.
Args:
str_action (str): String representation of an action to parse
direction (str): Limit actions searched through to just those that can run to this direction ("out", "in", "both")
logger (:obj:`logging.Logger`): a logger to log with
Returns:
:obj:`action.Action`: A parsed action object
"""
# Collect all viable actions that can run for each respective direction
outs = Action.get_actions("out")

@ -1,7 +1,18 @@
from actions.action import Action
class DropAction(Action):
"""
Geneva action to drop the given packet.
"""
frequency = 1
def __init__(self, environment_id=None):
"""
Initializes this drop action.
Args:
environment_id (str, optional): Environment ID of the strategy we are a part of
"""
Action.__init__(self, "drop", "both")
self.terminal = True
self.branching = False

@ -2,6 +2,10 @@ from actions.action import Action
class DuplicateAction(Action):
"""
Defines the DuplicateAction - returns two copies of the given packet.
"""
frequency = 3
def __init__(self, environment_id=None):
Action.__init__(self, "duplicate", "out")
self.branching = True
@ -13,3 +17,9 @@ class DuplicateAction(Action):
"""
logger.debug(" - Duplicating given packet %s" % str(packet))
return packet, packet.copy()
def mutate(self, environment_id=None):
"""
Swaps its left and right child
"""
self.left, self.right = self.right, self.left

@ -5,19 +5,32 @@ import actions.packet
from scapy.all import IP, TCP, fragment
MAX_UINT = 4294967295
class FragmentAction(Action):
def __init__(self, environment_id=None, correct_order=None, fragsize=-1, segment=True):
'''
correct_order specifies if the fragmented packets should come in the correct order
fragsize specifies how
'''
"""
Defines the FragmentAction for Geneva - fragments or segments the given packet.
"""
frequency = 2
def __init__(self, environment_id=None, correct_order=None, fragsize=-1, segment=True, overlap=0):
"""
Initializes a fragment action object.
Args:
environment_id (str, optional): Environment ID of the strategy this object is a part of
correct_order (bool, optional): Whether or not the fragments/segments should be returned in the correct order
fragsize (int, optional): The index this packet should be cut. Defaults to -1, which cuts it in half.
segment (bool, optional): Whether we should perform fragmentation or segmentation
overlap (int, optional): How many bytes the fragments/segments should overlap
"""
Action.__init__(self, "fragment", "out")
self.enabled = True
self.branching = True
self.terminal = False
self.fragsize = fragsize
self.segment = segment
self.overlap = overlap
if correct_order == None:
self.correct_order = self.get_rand_order()
else:
@ -87,6 +100,9 @@ class FragmentAction(Action):
Segments a packet into two, given the size of the first packet (0:fragsize)
Always returns two packets, since fragment is a branching action, so if we
are unable to segment, it will duplicate the packet.
If overlap is specified, it will select n bytes from the second packet
and append them to the first, and increment the sequence number accordingly
"""
if not packet.haslayer("TCP") or not hasattr(packet["TCP"], "load") or not packet["TCP"].load:
return packet, packet.copy() # duplicate if no TCP or no payload to segment
@ -101,7 +117,11 @@ class FragmentAction(Action):
fragsize = int(len(payload)/2)
# Craft new packets
pkt1 = IP(packet["IP"])/payload[:fragsize]
# Make sure we don't go out of bounds by choosing the min
overlap_bytes = min(len(payload[fragsize:]), self.overlap)
# Attach these bytes to the first packet
pkt1 = IP(packet["IP"])/payload[:fragsize + overlap_bytes]
pkt2 = IP(packet["IP"])/payload[fragsize:]
# We cannot rely on scapy's native parsing here - if a previous action has changed the
@ -116,7 +136,11 @@ class FragmentAction(Action):
packet2 = actions.packet.Packet(pkt2)
# Reset packet2's SYN number
packet2["TCP"].seq += fragsize
if packet2["TCP"].seq + fragsize > MAX_UINT:
# Wrap sequence numbers around if greater than MAX_UINT
packet2["TCP"].seq = packet2["TCP"].seq + fragsize - MAX_UINT - 1
else:
packet2["TCP"].seq += fragsize
del packet1["IP"].chksum
del packet2["IP"].chksum
@ -147,10 +171,14 @@ class FragmentAction(Action):
Returns a string representation with the fragsize
"""
s = Action.__str__(self)
if not self.overlap:
ending = "}"
else:
ending = ":" + str(self.overlap) + "}"
if self.segment:
s += "{" + "tcp" + ":" + str(self.fragsize) + ":" + str(self.correct_order) + "}"
s += "{" + "tcp" + ":" + str(self.fragsize) + ":" + str(self.correct_order) + ending
else:
s += "{" + "ip" + ":"+ str(self.fragsize) + ":" + str(self.correct_order) + "}"
s += "{" + "ip" + ":"+ str(self.fragsize) + ":" + str(self.correct_order) + ending
return s
def parse(self, string, logger):
@ -169,22 +197,36 @@ class FragmentAction(Action):
num_parameters = string.count(":")
# If num_parameters is greater than 2, it's not a valid fragment action
if num_parameters != 2:
msg = "Cannot parse fragment action %s" % string
logger.error(msg)
raise Exception(msg)
else:
if num_parameters == 2:
params = string.split(":")
seg, fragsize, correct_order = params
overlap = 0
if "tcp" in seg:
self.segment = True
else:
self.segment = False
elif num_parameters == 3:
params = string.split(":")
seg, fragsize, correct_order, overlap = params
if overlap.endswith("}"):
overlap = overlap[:-1] # Chop off trailing }
if "tcp" in seg:
self.segment = True
else:
self.segment = False
else:
msg = "Cannot parse fragment action %s" % string
logger.error(msg)
raise Exception(msg)
try:
# Try to convert to int
self.fragsize = int(fragsize)
except ValueError:
self.overlap = int(overlap)
except ValueError as e:
print(e)
msg = "Cannot parse fragment action %s" % string
logger.error(msg)
raise Exception(msg)
@ -196,3 +238,33 @@ class FragmentAction(Action):
self.correct_order = False
return True
def mutate(self, environment_id=None):
"""
Mutates the fragment action - it either chooses a new segment offset,
switches the packet order, and/or changes whether it segments or fragments.
"""
self.correct_order = self.get_rand_order()
self.segment = random.choice([True, True, True, False])
if self.segment:
if random.random() < 0.5:
self.fragsize = int(random.uniform(1, 60))
else:
self.fragsize = -1
else:
if random.random() < 0.2:
self.fragsize = int(random.uniform(1, 50))
else:
self.fragsize = -1
if random.random() < .5:
# Somewhat aggressively overlap
if random.random() < .5:
if self.fragsize == -1:
self.overlap = 5
else:
self.overlap = int(self.fragsize/2)
else:
self.overlap = int(random.uniform(1, 50))
return self

@ -1,4 +1,5 @@
import binascii
import copy
import random
import string
import os
@ -6,7 +7,6 @@ import urllib.parse
from scapy.all import IP, RandIP, UDP, DNS, DNSQR, Raw, TCP, fuzz
class Layer():
"""
Base class defining a Geneva packet layer.
@ -180,8 +180,7 @@ class Layer():
value = urllib.parse.unquote(value)
value = value.encode('utf-8')
# Add support for injecting arbitrary protocol payloads if requested
dns_payload = b"\x009ib\x81\x80\x00\x01\x00\x01\x00\x00\x00\x01\x08examples\x03com\x00\x00\x01\x00\x01\xc0\x0c\x00\x01\x00\x01\x00\x00\x01+\x00\x04\xc7\xbf2I\x00\x00)\x02\x00\x00\x00\x00\x00\x00\x00"
dns_payload = b"\x009ib\x81\x80\x00\x01\x00\x01\x00\x00\x00\x01\x08faceface\x03com\x00\x00\x01\x00\x01\xc0\x0c\x00\x01\x00\x01\x00\x00\x01+\x00\x04\xc7\xbf2I\x00\x00)\x02\x00\x00\x00\x00\x00\x00\x00"
http_payload = b"GET / HTTP/1.1\r\nHost: www.example.com\r\n\r\n"
value = value.replace(b"__DNS_REQUEST__", dns_payload)
@ -195,7 +194,17 @@ class Layer():
as a field properly.
"""
load = ''.join([random.choice(string.ascii_lowercase + string.digits) for k in range(10)])
return urllib.parse.quote(load)
return random.choice(["", "__DNS_REQUEST__", "__HTTP_REQUEST__", urllib.parse.quote(load)])
class RawLayer(Layer):
"""
Defines an interface for the scapy Raw layer.
"""
name = "Raw"
protocol = Raw
_fields = []
fields = []
class IPLayer(Layer):
@ -273,7 +282,11 @@ class IPLayer(Layer):
Sets the flags field. There is a bug in scapy, if you retrieve an empty
flags field, it will return "", but you cannot set this value back.
To reproduce this bug:
.. code-block:: python
>>> setattr(IP(), "flags", str(IP().flags)) # raises a ValueError
To handle this case, this method converts empty string to zero so that
it can be safely stored.
"""
@ -400,9 +413,15 @@ class TCPLayer(Layer):
'dataofs' : self.gen_dataofs,
'flags' : self.gen_flags,
'chksum' : self.gen_chksum,
'options' : self.gen_options
'options' : self.gen_options,
'window' : self.gen_window
}
def gen_window(self, field):
"""
Generates a window size.
"""
return random.choice(range(10, 200, 10))
def gen_chksum(self, field):
"""

@ -110,11 +110,10 @@ class Packet():
"""
iter_packet = self.packet
while iter_packet:
if iter_packet.name.lower() == "raw":
return
parsed_layer = Packet.parse_layer(iter_packet)
if parsed_layer:
yield parsed_layer
if parsed_layer.name != "Raw":
yield parsed_layer
iter_packet = parsed_layer.get_next_layer()
else:
iter_packet = iter_packet.payload

@ -1,8 +1,21 @@
from actions.action import Action
class SleepAction(Action):
"""
Defines the SleepAction - causes the engine to pause before sending a packet.
"""
# Do not select the sleep action during evolutions
frequency = 0
def __init__(self, time=1, environment_id=None):
Action.__init__(self, "sleep", "both")
"""
Initializes the sleep action.
Args:
time (float): How much time the packet should delay before sending
environment_id (str, optional): Environment ID of the strategy this action is a part of
"""
Action.__init__(self, "sleep", "out")
self.terminal = False
self.branching = False
self.time = time

@ -0,0 +1,85 @@
import threading
import os
import actions.packet
from scapy.all import sniff
from scapy.utils import PcapWriter
class Sniffer():
"""
The sniffer class lets the user begin and end sniffing whenever in a given location with a port to filter on.
Call start_sniffing to begin sniffing and stop_sniffing to stop sniffing.
"""
def __init__(self, location, port, logger):
"""
Intializes a sniffer object.
Needs a location and a port to filter on.
"""
self.stop_sniffing_flag = False
self.location = location
self.port = port
self.pcap_thread = None
self.packet_dumper = None
self.logger = logger
full_path = os.path.dirname(location)
assert port, "Need to specify a port in order to launch a sniffer"
if not os.path.exists(full_path):
os.makedirs(full_path)
def __packet_callback(self, scapy_packet):
"""
This callback is called whenever a packet is applied.
Returns true if it should finish, otherwise, returns false.
"""
packet = actions.packet.Packet(scapy_packet)
for proto in ["TCP", "UDP"]:
if(packet.haslayer(proto) and ((packet[proto].sport == self.port) or (packet[proto].dport == self.port))):
break
else:
return self.stop_sniffing_flag
self.logger.debug(str(packet))
self.packet_dumper.write(scapy_packet)
return self.stop_sniffing_flag
def __spawn_sniffer(self):
"""
Saves pcaps to a file. Should be run as a thread.
Ends when the stop_sniffing_flag is set. Should not be called by user
"""
self.packet_dumper = PcapWriter(self.location, append=True, sync=True)
while(self.stop_sniffing_flag == False):
sniff(stop_filter=self.__packet_callback, timeout=1)
def start_sniffing(self):
"""
Starts sniffing. Should be called by user.
"""
self.stop_sniffing_flag = False
self.pcap_thread = threading.Thread(target=self.__spawn_sniffer)
self.pcap_thread.start()
self.logger.debug("Sniffer starting to port %d" % self.port)
def __enter__(self):
"""
Defines a context manager for this sniffer; simply starts sniffing.
"""
self.start_sniffing()
return self
def __exit__(self, exc_type, exc_value, tb):
"""
Defines exit context manager behavior for this sniffer; simply stops sniffing.
"""
self.stop_sniffing()
def stop_sniffing(self):
"""
Stops the sniffer by setting the flag and calling join
"""
if(self.pcap_thread):
self.stop_sniffing_flag = True
self.pcap_thread.join()
self.logger.debug("Sniffer stopping")

@ -8,6 +8,8 @@ class Strategy(object):
def __init__(self, in_actions, out_actions, environment_id=None):
self.in_actions = in_actions
self.out_actions = out_actions
self.descendents = []
self.in_enabled = True
self.out_enabled = True
@ -52,6 +54,48 @@ class Strategy(object):
rep += "%s\n" % action_tree.pretty_print()
return rep
def initialize(self, logger, num_in_trees, num_out_trees, num_in_actions, num_out_actions, seed, disabled=None):
"""
Initializes a new strategy object randomly.
"""
# Disable specific forests if none requested
if num_in_trees == 0:
self.in_enabled = False
if num_out_trees == 0:
self.out_enabled = False
# If a specific population seed is requested, build using that
if seed:
starting_strat = actions.utils.parse(seed, logger)
self.out_actions = starting_strat.out_actions
self.in_actions = starting_strat.in_actions
return self
self.init_from_scratch(num_in_trees, num_out_trees, num_in_actions, num_out_actions, disabled=disabled)
return self
def init_from_scratch(self, num_in_trees, num_out_trees, num_in_actions, num_out_actions, disabled=None):
"""
Initializes this individual by drawing random actions.
"""
for _ in range(0, num_in_trees):
# Define a new in action tree
in_tree = actions.tree.ActionTree("in")
# Initialize the in tree
in_tree.initialize(num_in_actions, self.environment_id, disabled=disabled)
# Add them to this strategy
self.in_actions.append(in_tree)
for _ in range(0, num_out_trees):
# Define a new out action tree
out_tree = actions.tree.ActionTree("out")
# Initialize the out tree
out_tree.initialize(num_out_actions, self.environment_id, disabled=disabled)
# Add them to this strategy
self.out_actions.append(out_tree)
def act_on_packet(self, packet, logger, direction="out"):
"""
Runs the strategy on a given scapy packet.
@ -62,6 +106,7 @@ class Strategy(object):
return [packet]
return self.run_on_packet(packet, logger, direction)
def run_on_packet(self, packet, logger, direction):
"""
Runs the strategy on a given packet given the forest direction.
@ -86,3 +131,99 @@ class Strategy(object):
if not ran:
packets_to_send = [packet]
return packets_to_send
def mutate_dir(self, trees, direction, logger):
"""
Mutates a list of trees. Requires the direction the tree operates on
(in or out).
"""
pick = random.uniform(0, 1)
if pick < 0.1 or not trees:
new_tree = actions.tree.ActionTree(direction)
new_tree.initialize(1, self.environment_id)
trees.append(new_tree)
elif pick < 0.2 and trees:
trees.remove(random.choice(trees))
elif pick < 0.25 and trees and len(trees) > 1:
random.shuffle(trees)
else:
for action_tree in trees:
action_tree.mutate()
def mutate(self, logger):
"""
Top level mutation function for a strategy. Simply mutates the out
and in trees.
"""
if self.in_enabled:
self.mutate_dir(self.in_actions, "in", logger)
if self.out_enabled:
self.mutate_dir(self.out_actions, "out", logger)
return self
def swap_one(forest1, forest2):
"""
Swaps a random tree from forest1 and forest2.
It picks a random element within forest1 and a random element within forest2,
chooses a random index within each forest, and inserts the random element
"""
assert type(forest1) == list
assert type(forest2) == list
rand_idx1, rand_idx2 = 0, 0
donation, other_donation = None, None
if forest1:
donation = random.choice(forest1)
forest1.remove(donation)
if len(forest1) > 0:
rand_idx1 = random.choice(list(range(0, len(forest1))))
if forest2:
other_donation = random.choice(forest2)
forest2.remove(other_donation)
if len(forest2) > 0:
rand_idx2 = random.choice(list(range(0, len(forest2))))
if other_donation:
forest1.insert(rand_idx1, other_donation)
if donation:
forest2.insert(rand_idx2, donation)
return True
def do_mate(forest1, forest2):
"""
Performs mating between two given forests (lists of trees).
With 80% probability, a random tree from each forest are mated,
otherwise, a random tree is swapped between them.
"""
# If 80% and there are trees in both forests to mate, or
# if there is only 1 tree in each forest, mate those trees
if (random.random() < 0.8 and forest1 and forest2) or \
(len(forest1) == 1 and len(forest2) == 1):
tree1 = random.choice(forest1)
tree2 = random.choice(forest2)
return tree1.mate(tree2)
# Otherwise, swap a random tree from each forest
elif forest1 or forest2:
return swap_one(forest1, forest2)
return False
def mate(ind1, ind2, indpb):
"""
Executes a uniform crossover that modify in place the two
individuals. The attributes are swapped according to the
*indpb* probability.
"""
out_success, in_success = True, True
if ind1.out_enabled and random.random() < indpb:
out_success = do_mate(ind1.out_actions, ind2.out_actions)
if ind1.in_enabled and random.random() < indpb:
in_success = do_mate(ind1.in_actions, ind2.in_actions)
return out_success and in_success

@ -3,11 +3,11 @@ TamperAction
One of the four packet-level primitives supported by Geneva. Responsible for any packet-level
modifications (particularly header modifications). It supports the following primitives:
- no operation: it returns the packet given
- replace: it changes a packet field to a fixed value
- corrupt: it changes a packet field to a randomly generated value each time it is run
- add: adds a given value to the value in a field
- compress: performs DNS decompression on the packet (if applicable)
- no operation: it returns the packet given
- replace: it changes a packet field to a fixed value
- corrupt: it changes a packet field to a randomly generated value each time it is run
- add: adds a given value to the value in a field
- compress: performs DNS decompression on the packet (if applicable)
"""
from actions.action import Action
@ -20,18 +20,72 @@ import random
# All supported tamper primitives
SUPPORTED_PRIMITIVES = ["corrupt", "replace", "add", "compress"]
# Tamper primitives we can mutate to by default
ACTIVATED_PRIMITIVES = ["replace", "corrupt", "add"]
class TamperAction(Action):
"""
Defines the TamperAction for Geneva.
"""
frequency = 5
def __init__(self, environment_id=None, field=None, tamper_type=None, tamper_value=None, tamper_proto="TCP"):
"""
Creates a tamper object.
Args:
environment_id (str, optional): environment_id of a previously run strategy, used to find packet captures
field (str, optional): field that the object will tamper. If not set, all the parameters are chosen randomly
tamper_type (str, optional): primitive this tamper will use ("corrupt")
tamper_value (str, optional): value to tamper to
tamper_proto (str, optional): protocol we are tampering
"""
Action.__init__(self, "tamper", "both")
self.field = field
self.tamper_value = tamper_value
self.tamper_proto = actions.utils.string_to_protocol(tamper_proto)
self.tamper_proto_str = tamper_proto
self.tamper_type = tamper_type
if not self.tamper_type:
self._mutate_tamper_type()
if not self.field:
self._mutate(environment_id)
def mutate(self, environment_id=None):
"""
Mutate can switch between the tamper type, field.
"""
# With some probability switch tamper types
pick = random.random()
if pick < 0.2:
self._mutate_tamper_type()
else:
self._mutate(environment_id)
def _mutate_tamper_type(self):
"""
Randomly picks a tamper type to change to.
"""
self.tamper_type = random.choice(ACTIVATED_PRIMITIVES)
if self.tamper_type == "compress":
self.tamper_proto_str = "DNS"
self.tamper_proto = actions.utils.string_to_protocol(self.tamper_proto_str)
self.field = "qd"
def _mutate(self, environment_id):
"""
Mutates this action using:
- previously seen packets with 50% probability
- a fuzzed packet with 50% probability
"""
# Retrieve a new protocol and field options for this protocol
proto, field, value = actions.utils.get_from_fuzzed_or_real_packet(environment_id, 0.5)
self.tamper_proto = proto
self.tamper_proto_str = proto.__name__
self.field = field
self.tamper_value = value
def tamper(self, packet, logger):
"""

@ -15,8 +15,19 @@ class TraceAction(Action):
TraceAction is an experimental action that is never used
in actual evolution
"""
# Do not select Trace during evolutions
frequency = 0
def __init__(self, start_ttl=1, end_ttl=64, environment_id=None):
"""
Initializes the trace action.
Args:
start_ttl (int): Starting TTL to use
end_ttl (int): TTL to end with
environment_id (str, optional): Environment ID associated with the strategy we are a part of
"""
Action.__init__(self, "trace", "out")
self.enabled = True
self.terminal = True
self.branching = False
self.start_ttl = start_ttl
@ -40,7 +51,7 @@ class TraceAction(Action):
return packet, None
if self.ran:
logger.debug(" - trace action already ran. Dropping given traffic. To reset this action, restart the engine.")
logger.debug(" - trace action already ran. Dropping given traffic.")
return None, None
self.ran = True

@ -24,12 +24,35 @@ class ActionTree():
"""
def __init__(self, direction, trigger=None):
"""
Creates this action tree.
Args:
direction (str): Direction this tree is facing ("out", "in")
trigger (:obj:`actions.trigger.Trigger`): Trigger to use with this tree
"""
self.trigger = trigger
self.action_root = None
self.direction = direction
self.environment_id = None
self.ran = False
def initialize(self, num_actions, environment_id, allow_terminal=True, disabled=None):
"""
Sets up this action tree with a given number of random actions.
Note that the returned action trees may have less actions than num_actions
if terminal actions are used.
"""
self.environment_id = environment_id
self.trigger = actions.trigger.Trigger(None, None, None, environment_id=environment_id)
if not allow_terminal or random.random() > 0.1:
allow_terminal = False
for _ in range(num_actions):
new_action = self.get_rand_action(self.direction, disabled=disabled)
self.add_action(new_action)
return self
def __iter__(self):
"""
Sets up a preoder iterator for the tree.
@ -222,8 +245,8 @@ class ActionTree():
if not node.right:
yield right_packet
# If we have a left action and were given a packet to pass on, run
# on the left packet
# If we have a right action and were given a packet to pass on, run
# on the right packet
if node.right and right_packet:
for rpacket in self.do_run(node.right, right_packet, logger):
yield rpacket
@ -383,6 +406,22 @@ class ActionTree():
break
return action_added
def get_rand_action(self, direction, request=None, allow_terminal=True, disabled=None):
"""
Retrieves and initializes a random action that can run in the given direction.
"""
pick = random.random()
action_options = actions.action.Action.get_actions(direction, disabled=disabled, allow_terminal=allow_terminal)
# Check to make sure there are still actions available to use
assert action_options, "No actions were available"
act_dict = {}
all_opts = []
for action_name, act_cls in action_options:
act_dict[action_name] = act_cls
all_opts += ([act_cls] * act_cls.frequency)
new_action = act_dict.get(request, random.choice(all_opts))
return new_action(environment_id=self.environment_id)
def remove_one(self):
"""
Removes a random leaf from the tree.
@ -392,6 +431,26 @@ class ActionTree():
action = random.choice(self)
return self.remove_action(action)
def mutate(self):
"""
Mutates this action tree with respect to a given direction.
"""
pick = random.uniform(0, 1)
if pick < 0.20 or not self.action_root:
new_action = self.get_rand_action(direction=self.direction)
self.add_action(new_action)
elif pick < 0.65 and self.action_root:
action = random.choice(self)
action.mutate(environment_id=self.environment_id)
# If this individual has never been run under the evaluator,
# or if it ran and it failed, it won't have an environment_id,
# which means it has no saved packets to read from.
elif pick < 0.80 and self.environment_id:
self.trigger.mutate(self.environment_id)
else:
self.remove_one()
return self
def choose_one(self):
"""
Picks a random element in the tree.
@ -415,6 +474,40 @@ class ActionTree():
return action, "right"
return None, None
def swap(self, my_donation, other_tree, other_donation):
"""
Swaps a node in this tree with a node in another tree.
"""
parent, direction = self.get_parent(my_donation)
other_parent, other_direction = other_tree.get_parent(other_donation)
# If this tree is empty or I'm trying to donate my root
if not my_donation or not parent:
parent = self
direction = "action_root"
# if the other tree is empty or they are trying to donate their root
if not other_donation or not other_parent:
other_parent = other_tree
other_direction = "action_root"
setattr(parent, direction, other_donation)
setattr(other_parent, other_direction, my_donation)
return True
def mate(self, other_tree):
"""
Mates this tree with another tree.
"""
# If both trees are empty, nothing to do
if not self.action_root and not other_tree.action_root:
return False
# Chose an action node in this tree to swap
my_swap_node = self.choose_one()
other_swap_node = other_tree.choose_one()
return self.swap(my_swap_node, other_tree, other_swap_node)
def pretty_print_help(self, root, visual=False, parent=None):
"""
Pretty prints the tree.
@ -448,6 +541,7 @@ class ActionTree():
return newroot
def pretty_print(self, visual=False):
"""
Pretty prints the tree.

@ -15,6 +15,7 @@ class Trigger(object):
- trigger_value: the value in the trigger_field that, upon a match, will cause the trigger to fire
- environment_id: environment_id the current trigger is running under. Used to retrieve previously saved packets
- gas: how many times this trigger can fire before it stops triggering. gas=None disables gas (unlimited triggers.)
- has_wildcard: represents if the trigger will match a specific value, or any value containing trigger_value
"""
self.trigger_type = trigger_type
self.trigger_field = trigger_field
@ -23,9 +24,55 @@ class Trigger(object):
self.environment_id = environment_id
self.num_seen = 0
self.gas_remaining = gas
self.has_wildcard = False
# Bomb triggers act like reverse triggers. They run the action only after the action has been triggered x times
self.bomb_trigger = bool(gas and gas < 0)
self.ran = False
# ignore numerical trigger values
if isinstance(self.trigger_value, (str)):
# check if value field is wildcarded or not
if(len(self.trigger_value) != 0 and self.trigger_value[-1] == '*'):
self.has_wildcard = True
# remove '*' wildcard from trigger_value for ease of use
self.trigger_value = self.trigger_value[:-1]
if not self.trigger_type:
self.trigger_type, self.trigger_proto, self.trigger_field, self.trigger_value, self.gas_remaining = Trigger.get_rand_trigger(environment_id, 1)
@staticmethod
def get_gas():
"""
Returns a random value for gas for this trigger.
"""
if GAS_ENABLED and random.random() < 0.2:
# Use gas in 20% of scenarios
# Pick a number for gas between 0 - 5
gas_remaining = int(random.random() * 5)
else:
# Do not use gas
gas_remaining = None
return gas_remaining
@staticmethod
def get_rand_trigger(environment_id, real_packet_probability):
"""
Creates a random trigger.
"""
proto, field, value = actions.utils.get_from_fuzzed_or_real_packet(environment_id, real_packet_probability, enable_options=False, enable_load=False)
gas_remaining = Trigger.get_gas()
if not FIXED_TRIGGER:
# Only "field" triggers are supported currently
return "field", proto.__name__, field, value, gas_remaining
return (FIXED_TRIGGER.trigger_type,
FIXED_TRIGGER.trigger_proto,
FIXED_TRIGGER.trigger_field,
FIXED_TRIGGER.trigger_value,
FIXED_TRIGGER.gas_remaining)
def mutate(self, environment_id, real_packet_probability=0.5):
"""
Mutates this trigger object by picking a new protocol, field, and value.
"""
self.trigger_type, self.trigger_proto, self.trigger_field, self.trigger_value, self.gas_remaining = Trigger.get_rand_trigger(environment_id, real_packet_probability)
def is_applicable(self, packet, logger):
"""
@ -37,7 +84,10 @@ class Trigger(object):
return False
packet_value = packet.get(self.trigger_proto, self.trigger_field)
will_run = (self.trigger_value == packet_value)
if self.has_wildcard:
will_run = (self.trigger_value in packet_value)
else:
will_run = (self.trigger_value == packet_value)
# Track if this action is used
if (not GAS_ENABLED or self.gas_remaining is None) and will_run:

@ -2,6 +2,7 @@ import copy
import datetime
import importlib
import inspect
import json
import logging
import os
import string
@ -12,24 +13,38 @@ import urllib.parse
import actions.action
import actions.trigger
import actions.packet
import plugins.plugin_client
import plugins.plugin_server
from scapy.all import TCP, IP, UDP, rdpcap
import netifaces
RUN_DIRECTORY = datetime.datetime.now().strftime("%Y-%m-%d_%H:%M:%S")
RUN_DIRECTORY = os.path.join("trials", datetime.datetime.now().strftime("%Y-%m-%d_%H:%M:%S"))
# Hard coded options
FLAGFOLDER = "flags"
# Holds copy of console file handler's log level
CONSOLE_LOG_LEVEL = logging.DEBUG
CONSOLE_LOG_LEVEL = "debug"
BASEPATH = os.path.dirname(os.path.abspath(__file__))
PROJECT_ROOT = os.path.dirname(BASEPATH)
class SkipStrategyException(Exception):
"""
Raised to signal that this strategy evaluation should be cut off.
"""
def __init__(self, msg, fitness):
"""
Creates the exception with the fitness to pass back
"""
self.fitness = fitness
self.msg = msg
def parse(requested_trees, logger):
"""
Parses a string representation of a solution into its object form.
@ -71,7 +86,9 @@ def parse(requested_trees, logger):
# strategies, so restore the "|" that was lost from the split
str_action = str_action + "|"
new_tree = actions.tree.ActionTree(direction)
new_tree.parse(str_action, logger)
success = new_tree.parse(str_action, logger)
if success is False:
raise actions.tree.ActionTreeParseError("Failed to parse tree")
# Once all the actions are parsed, add this tree to the
# current direction of actions
@ -85,7 +102,7 @@ def parse(requested_trees, logger):
return strat
def get_logger(basepath, log_dir, logger_name, log_name, environment_id, log_level=logging.DEBUG):
def get_logger(basepath, log_dir, logger_name, log_name, environment_id, log_level="DEBUG"):
"""
Configures and returns a logger.
"""
@ -100,7 +117,7 @@ def get_logger(basepath, log_dir, logger_name, log_name, environment_id, log_lev
os.makedirs(flag_path)
# Set up a client logger
logger = logging.getLogger(logger_name + environment_id)
logger.setLevel(logging.DEBUG)
logger.setLevel("DEBUG")