Gamercat
/
geneva
mirror of https://github.com/Kkevsterrr/geneva
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

Initial commit

Browse Source 
Initial commit of Geneva
pull/1/head
George Hughey 4 years ago
commit d0fd49800c
19 changed files with 2943 additions and 0 deletions
Show Stats Download Patch File Download Diff File

115
README.md
Unescape Escape View File

@ -0,0 +1,115 @@
# Geneva
Geneva is a artificial intelligence tool that defeats censorship by exploiting bugs in censors, such as those in China, India, and Kazakhstan. Unlike many other anti-censorship solutions which require assistance from outside the censoring regime (Tor, VPNs, etc.), Geneva runs strictly on the client.
Under the hood, Geneva uses a genetic algorithm to evolve censorship evasion strategies and has found several previously unknown bugs in censors. Geneva's strategies manipulate the client's packets to confuse the censor without impacting the client/server communication. This makes Geneva effective against many types of in-network censorship (though it cannot be used against IP-blocking censorship).
This code release specifically contains the strategy engine used by Geneva, its Python API, and a subset of published strategies, so users and researchers can test and deploy Geneva's strategies. To learn more about how Geneva works, visit [How it Works](#How-it-Works). We will be releasing the genetic algorithm at a later date.
## Setup
Geneva has been developed and tested for Centos or Debian-based systems. Due to limitations of
netfilter and raw sockets, Geneva does not work on OS X or Windows at this time.
Install netfilterqueue dependencies:
```
# sudo apt-get install build-essential python-dev libnetfilter-queue-dev libffi-dev libssl-dev
```
Install Python dependencies:
```
# python3 -m pip install -r requirements.txt
```
## Running it
```
# python3 engine.py --server-port 80 --strategy "[TCP:flags:A]-duplicate(,tamper{TCP:flags:replace:R})-| \/" --log debug
2019-10-14 16:34:45 DEBUG:[ENGINE] Engine created with strategy \/ (ID bm3kdw3r) to port 80
2019-10-14 16:34:45 DEBUG:[ENGINE] Configuring iptables rules
2019-10-14 16:34:45 DEBUG:[ENGINE] iptables -A OUTPUT -p tcp --sport 80 -j NFQUEUE --queue-num 1
2019-10-14 16:34:45 DEBUG:[ENGINE] iptables -A INPUT -p tcp --dport 80 -j NFQUEUE --queue-num 2
2019-10-14 16:34:45 DEBUG:[ENGINE] iptables -A OUTPUT -p udp --sport 80 -j NFQUEUE --queue-num 1
2019-10-14 16:34:45 DEBUG:[ENGINE] iptables -A INPUT -p udp --dport 80 -j NFQUEUE --queue-num 2
```
Note that if you have stale `iptables` rules or other rules that rely on Geneva's default queues,
this will fail. To fix this, remove those rules.
## Strategy Library
Geneva has found dozens of strategies that work against censors in China, Kazakhstan, and India. We include several of these strategies in [strategies.md](strategies.md). Note that this file contains success rates for each individual country; a strategy that works in one country may not work as well as other countries.
Researchers have observed that strategies may have differing success rates based on your exact location. Although we have not observed this from our vantage points, you may find that some strategies may work differently in a country we have tested. If this is the case, don't be alarmed. However, please feel free to reach out to a member of the team directly or open an issue on this page so we can track how the strategies work from other geographic locations.
## Disclaimer
Running these strategies may place you at risk if you use it within a censoring regime. Geneva takes overt actions that interfere with the normal operations of a censor and its strategies are detectable on the network. Geneva is not an anonymity tool, nor does it encrypt any traffic. Understand the risks of running Geneva in your country before trying it.
-------
## How it Works
See our paper for an in-depth read on how Geneva works. Below is a rundown of the format of Geneva's strategy DNA.
### Strategy DNA
Geneva's strategies can be arbitrarily complicated, and it defines a well-formatted syntax for
expressing strategies to the engine.
A strategy is simply a _description of how network traffic should be modified_. A strategy is not
code, it is a description that tells the engine how it should operate over traffic.
A strategy divides how it handles outbound and inbound packets: these are separated in the DNA by a
"\\/". Specifically, the strategy format is `<outbound forest> \/ <inbound forest>`. If `\/` is not
present in a strategy, all of the action trees are in the outbound forest.
Both forests are composed of action trees, and each forest is allowed an arbitrarily many trees.
An action tree is comprised of a _trigger_ and a _tree_. The trigger describes _when_ the strategy
should run, and the tree describes what should happen when the trigger fires. Recall that Geneva
operates at the packet level, therefore all triggers are packet-level triggers. Action trees start
with a trigger, and always end with a `-|`.
Triggers operate as exact-matches, are formatted as follows: `[<protocol>:<field>:<value>]`. For
example, the trigger: `[TCP:flags:S]` will run its corresponding tree whenever it sees a `SYN`
TCP packet. If the corresponding action tree is `[TCP:flags:S]-drop-|`, this action tree will cause
the engine to drop any `SYN` packets. `[TCP:flags:S]-duplicate-|` will cause the engine to
duplicate the SYN packet.
Depending on the type of action, some actions can have up to two children. These are represented
with the following syntax: `[TCP:flags:S]-duplicate(<left_child>,<right_child>)-|`, where
`<left_child>` and `<right_child>` themselves are trees. If `(,)` is not specified, any packets
that emerge from the action will be sent on the wire.
Any action that has parameters associated with it contain those parameters in `{}`. Consider the
following strategy with `tamper`.
```
[TCP:flags:A]-duplicate(tamper{TCP:flags:replace:R},)-| \/
```
This strategy takes outbound `ACK` packets and duplicates them. To the first duplicate, it tampers
the packet by replacing the `TCP` `flags` field with `RST`, and does nothing to the second
duplicate.
Note that due to NFQueue limitations, actions that introduce branching (fragment, duplicate) are
disabled for incoming action forests.
-------
## Citation
If you like the work or plan to use it in your projects, please follow the guidelines in [citation.bib](https://github.com/Kkevsterrr/geneva/blob/master/citation.bib).
## Paper
See [our paper](http://geneva.cs.umd.edu/papers/geneva_ccs19.pdf) from CCS for an in-depth dive into how it works.
## Contributors
[Kevin Bock](https://github.com/Kkevsterrr)
[George Hughey](https://github.com/ecthros)
[Xiao Qiang](https://twitter.com/rockngo)
[Dave Levin](https://www.cs.umd.edu/~dml/)

149
actions/action.py
Unescape Escape View File

@ -0,0 +1,149 @@
"""
Action
Geneva object for defining a packet-level action.
"""
import inspect
import importlib
import os
import sys
import actions.utils
ACTION_CACHE = {}
ACTION_CACHE["in"] = {}
ACTION_CACHE["out"] = {}
BASEPATH = os.path.sep.join(os.path.dirname(os.path.abspath(__file__)).split(os.path.sep)[:-1])
class Action():
"""
Defines the superclass for a Geneva Action.
"""
# Give each Action a unique ID - this is needed for graphing/visualization
ident = 0
def __init__(self, action_name, direction):
"""
Initializes this action object.
"""
self.enabled = True
self.action_name = action_name
self.direction = direction
self.requires_undo = False
self.num_seen = 0
self.left = None
self.right = None
self.branching = False
self.terminal = False
self.ident = Action.ident
Action.ident += 1
def applies(self, direction):
"""
Returns whether this action applies to the given direction, as
branching actions are not supported on inbound trees.
"""
if direction == self.direction or self.direction == "both":
return True
return False
def mutate(self, environment_id=None):
"""
Mutates packet.
"""
def __str__(self):
"""
Defines string representation of this action.
"""
return "%s" % (self.action_name)
@staticmethod
def get_actions(direction, disabled=None, allow_terminal=True):
"""
Dynamically imports all of the Action classes in this directory.
Will only return terminal actions if terminal is set to True.
"""
if disabled is None:
disabled = []
# Recursively call this function again to enumerate in and out actions
if direction.lower() == "both":
return list(set(Action.get_actions("in", disabled=disabled, allow_terminal=allow_terminal) + \
Action.get_actions("out", disabled=disabled, allow_terminal=allow_terminal)))
terminal = "terminal"
if not allow_terminal:
terminal = "non-terminal"
if terminal not in ACTION_CACHE[direction]:
ACTION_CACHE[direction][terminal] = {}
else:
return ACTION_CACHE[direction][terminal]
collected_actions = []
# Get the base path for the project relative to this file
path = os.path.join(BASEPATH, "actions")
for action_file in os.listdir(path):
if not action_file.endswith(".py"):
continue
action = action_file.replace(".py", "")
if BASEPATH not in sys.path:
sys.path.append(BASEPATH)
importlib.import_module("actions." + action)
def check_action(obj):
return inspect.isclass(obj) and \
issubclass(obj, actions.action.Action) and \
obj != actions.action.Action and \
obj().applies(direction) and \
obj().enabled and \
not any([x in str(obj) for x in disabled]) and \
(allow_terminal or not obj().terminal)
clsmembers = inspect.getmembers(sys.modules["actions."+action], predicate=check_action)
collected_actions += clsmembers
collected_actions = list(set(collected_actions))
ACTION_CACHE[direction][terminal] = collected_actions
return collected_actions
@staticmethod
def parse_action(str_action, direction, logger):
"""
Parses a string action into the action object.
"""
# Collect all viable actions that can run for each respective direction
outs = Action.get_actions("out")
ins = Action.get_actions("in")
# If we're currently parsing the OUT forest, only search the out-compatible actions
if direction == "out":
search = outs
# Otherwise only search in-compatible actions (no branching)
else:
search = ins
action_obj = None
data = None
# If this action has parameters (defined within {} attached to the action),
# split off the data parameters from the raw action name
if "{" in str_action:
str_action, data = str_action.split("{")
data = data.replace("}", "")
# Search through all of the actions available for this direction to find the right class
for action_name, action_cls in search:
if str_action.strip() and str_action.lower() in action_name.lower():
# Define the action, and give it a reference to its parent strategy
action_obj = action_cls()
# If this action has data, ask the new module to parse & initialize itself to it
if data:
# Pass our logger to the action to alert us if it can't parse something
action_obj.parse(data, logger)
return action_obj

15
actions/drop.py
Unescape Escape View File

@ -0,0 +1,15 @@
from actions.action import Action
class DropAction(Action):
def __init__(self, environment_id=None):
Action.__init__(self, "drop", "both")
self.terminal = True
self.branching = False
def run(self, packet, logger):
"""
The drop action returns None for both it's left and right children, and
does not pass the packet along for continued use.
"""
logger.debug(" - Dropping given packet.")
return None, None

21
actions/duplicate.py
Unescape Escape View File

@ -0,0 +1,21 @@
from actions.action import Action
class DuplicateAction(Action):
def __init__(self, environment_id=None):
Action.__init__(self, "duplicate", "out")
self.branching = True
def run(self, packet, logger):
"""
The duplicate action duplicates the given packet and returns one copy
for the left branch, and one for the right branch.
"""
logger.debug(" - Duplicating given packet %s" % str(packet))
return packet, packet.copy()
def mutate(self, environment_id=None):
"""
Swaps its left and right child
"""
self.left, self.right = self.right, self.left

217
actions/fragment.py
Unescape Escape View File

@ -0,0 +1,217 @@
import random
from actions.action import Action
import actions.packet
from scapy.all import IP, TCP, fragment
class FragmentAction(Action):
def __init__(self, environment_id=None, correct_order=None, fragsize=-1, segment=True):
'''
correct_order specifies if the fragmented packets should come in the correct order
fragsize specifies how
'''
Action.__init__(self, "fragment", "out")
self.enabled = True
self.branching = True
self.terminal = False
self.fragsize = fragsize
self.segment = segment
if correct_order == None:
self.correct_order = self.get_rand_order()
else:
self.correct_order = correct_order
def get_rand_order(self):
"""
Randomly decides if the fragments should be reversed.
"""
return random.choice([True, False])
def fragment(self, original, fragsize):
"""
Fragments a packet into two, given the size of the first packet (0:fragsize)
Always returns two packets
"""
if fragsize == 0:
frags = [original]
else:
frags = fragment(original, fragsize=fragsize)
# If there were more than 2 fragments, join the loads so we still have 2 packets
if len(frags) > 2:
for frag in frags[2:]:
frags[1]["IP"].load += frag["IP"].load
# After scapy fragmentation, the flags field is set to "MF+DF"
# In order for the packet to remain valid, strip out the "MF"
frags[1]["IP"].flags = "DF"
# If scapy tried to fragment but there were only enough bytes for 1 packet, just duplicate it
elif len(frags) == 1:
frags.append(frags[0].copy())
return frags[0], frags[1]
def ip_fragment(self, packet, logger):
"""
Perform IP fragmentation.
"""
if not packet.haslayer("IP") or not hasattr(packet["IP"], "load"):
return packet, packet.copy() # duplicate if no TCP or no payload to segment
load = ""
if packet.haslayer("TCP"):
load = bytes(packet["TCP"])
elif packet.haslayer("UDP"):
load = bytes(packet["UDP"])
else:
load = bytes(packet["IP"].load)
# If there is no load, duplicate the packet
if not load:
return packet, packet.copy()
if self.fragsize == -1 or (self.fragsize * 8) > len(load) or len(load) <= 8:
fragsize = int(int(((int(len(load)/2))/8))*8)
frags = self.fragment(packet.copy().packet, fragsize=fragsize)
else:
# packet can be fragmented as requested
frags = self.fragment(packet.copy().packet, fragsize=self.fragsize*8)
packet1 = actions.packet.Packet(frags[0])
packet2 = actions.packet.Packet(frags[1])
if self.correct_order:
return packet1, packet2
else:
return packet2, packet1
def tcp_segment(self, packet, logger):
"""
Segments a packet into two, given the size of the first packet (0:fragsize)
Always returns two packets, since fragment is a branching action, so if we
are unable to segment, it will duplicate the packet.
"""
if not packet.haslayer("TCP") or not hasattr(packet["TCP"], "load") or not packet["TCP"].load:
return packet, packet.copy() # duplicate if no TCP or no payload to segment
# Get the original payload and delete it from the packet so it
# doesn't come along when copying the TCP layer
payload = packet["TCP"].load
del(packet["TCP"].load)
fragsize = self.fragsize
if self.fragsize == -1 or self.fragsize > len(payload) - 1:
fragsize = int(len(payload)/2)
# Craft new packets
pkt1 = IP(packet["IP"])/payload[:fragsize]
pkt2 = IP(packet["IP"])/payload[fragsize:]
# We cannot rely on scapy's native parsing here - if a previous action has changed the
# fragment offset, scapy will not identify this as TCP, so we must do it for scapy
if not pkt1.haslayer("TCP"):
pkt1 = IP(packet["IP"])/TCP(bytes(pkt1["IP"].load))
if not pkt2.haslayer("TCP"):
pkt2 = IP(packet["IP"])/TCP(bytes(pkt2["IP"].load))
packet1 = actions.packet.Packet(pkt1)
packet2 = actions.packet.Packet(pkt2)
# Reset packet2's SYN number
packet2["TCP"].seq += fragsize
del packet1["IP"].chksum
del packet2["IP"].chksum
del packet1["IP"].len
del packet2["IP"].len
del packet1["TCP"].chksum
del packet2["TCP"].chksum
del packet1["TCP"].dataofs
del packet2["TCP"].dataofs
if self.correct_order:
return [packet1, packet2]
else:
return [packet2, packet1]
def run(self, packet, logger):
"""
The fragment action fragments each given packet.
"""
logger.debug(" - Fragmenting given packet %s" % str(packet))
if self.segment:
return self.tcp_segment(packet, logger)
else:
return self.ip_fragment(packet, logger)
def __str__(self):
"""
Returns a string representation with the fragsize
"""
s = Action.__str__(self)
if self.segment:
s += "{" + "tcp" + ":" + str(self.fragsize) + ":" + str(self.correct_order) + "}"
else:
s += "{" + "ip" + ":"+ str(self.fragsize) + ":" + str(self.correct_order) + "}"
return s
def parse(self, string, logger):
"""
Parses a string representation of fragmentation. Nothing particularly special,
but it does check for a the fragsize.
Note that the given logger is a DIFFERENT logger than the logger passed
to the other functions, and they cannot be used interchangeably. This logger
is attached to the main GA driver, and is run outside the evaluator. When the
action is actually run, it's run within the evaluator, which by necessity must
pass in a different logger.
"""
# Count the number of params in this given string
num_parameters = string.count(":")
# If num_parameters is greater than 2, it's not a valid fragment action
if num_parameters != 2:
msg = "Cannot parse fragment action %s" % string
logger.error(msg)
raise Exception(msg)
else:
params = string.split(":")
seg, fragsize, correct_order = params
if "tcp" in seg:
self.segment = True
else:
self.segment = False
try:
# Try to convert to int
self.fragsize = int(fragsize)
except ValueError:
msg = "Cannot parse fragment action %s" % string
logger.error(msg)
raise Exception(msg)
# Parse ordering
if correct_order.startswith('True'):
self.correct_order = True
else:
self.correct_order = False
return True
def mutate(self, environment_id=None):
"""
Mutates the fragment action - it either chooses a new segment offset,
switches the packet order, and/or changes whether it segments or fragments.
"""
self.correct_order = self.get_rand_order()
self.segment = random.choice([True, True, True, False])
if self.segment:
if random.random() < 0.5:
self.fragsize = int(random.uniform(1, 60))
else:
self.fragsize = -1
else:
if random.random() < 0.2:
self.fragsize = int(random.uniform(1, 50))
else:
self.fragsize = -1
return self

594
actions/layer.py
Unescape Escape View File

@ -0,0 +1,594 @@
import binascii
import random
import string
import os
import urllib.parse
from scapy.all import IP, RandIP, UDP, Raw, TCP, fuzz
class Layer():
"""
Base class defining a Geneva packet layer.
"""
protocol = None
def __init__(self, layer):
"""
Initializes this layer.
"""
self.layer = layer
# No custom setter, getters, generators, or parsers are needed by default
self.setters = {}
self.getters = {}
self.generators = {}
self.parsers = {}
@classmethod
def reset_restrictions(cls):
"""
Resets field restrictions placed on this layer.
"""
cls.fields = cls._fields
def get_next_layer(self):
"""
Given the current layer returns the next layer beneath us.
"""
if len(self.layer.layers()) == 1:
return None
return self.layer[1]
def get_random(self):
"""
Retreives a random field and value.
"""
field = random.choice(self.fields)
return field, self.get(field)
def gen_random(self):
"""
Generates a random field and value.
"""
assert self.fields, "Layer %s doesn't have any fields" % str(self)
field = random.choice(self.fields)
return field, self.gen(field)
@classmethod
def name_matches(cls, name):
"""
Checks if given name matches this layer name.
"""
return name.upper() == cls.name.upper()
def get(self, field):
"""
Retrieves the value from a given field.
"""
assert field in self.fields
if field in self.getters:
return self.getters[field](field)
# Dual field accessors are fields that require two pieces of information
# to retrieve them (for example, "options-eol"). These are delimited by
# a dash "-".
base = field.split("-")[0]
if "-" in field and base in self.getters:
return self.getters[base](field)
return getattr(self.layer, field)
def set(self, packet, field, value):
"""
Sets the value for a given field.
"""
assert field in self.fields
base = field.split("-")[0]
if field in self.setters:
self.setters[field](packet, field, value)
# Dual field accessors are fields that require two pieces of information
# to retrieve them (for example, "options-eol"). These are delimited by
# a dash "-".
elif "-" in field and base in self.setters:
self.setters[base](packet, field, value)
else:
setattr(self.layer, field, value)
# Request the packet be reparsed to confirm the value is stable
# XXX Temporarily disabling the reconstitution check due to scapy bug (#2034)
#assert bytes(self.protocol(bytes(self.layer))) == bytes(self.layer)
def gen(self, field):
"""
Generates a value for this field.
"""
assert field in self.fields
if field in self.generators:
return self.generators[field](field)
# Dual field accessors are fields that require two pieces of information
# to retrieve them (for example, "options-eol"). These are delimited by
# a dash "-".
base = field.split("-")[0]
if "-" in field and base in self.generators:
return self.generators[base](field)
sample = fuzz(self.protocol())
new_value = getattr(sample, field)
if new_value == None:
new_value = 0
elif type(new_value) != int:
new_value = new_value._fix()
return new_value
def parse(self, field, value):
"""
Parses the given value for a given field. This is useful for fields whose
value cannot be represented in a string type easily - it lets us define
a common string representation for the strategy, and parse it back into
a real value here.
"""
assert field in self.fields
if field in self.parsers:
return self.parsers[field](field, value)
# Dual field accessors are fields that require two pieces of information
# to retrieve them (for example, "options-eol"). These are delimited by
# a dash "-".
base = field.split("-")[0]
if "-" in field and base in self.parsers:
return self.parsers[base](field, value)
try:
parsed = int(value)
except ValueError:
parsed = value
return parsed
def get_load(self, field):
"""
Helper method to retrieve load, as scapy doesn't recognize 'load' as
a regular field properly.
"""
try:
load = self.layer.payload.load
except AttributeError:
pass
try:
load = self.layer.load
except AttributeError:
return ""
if not load:
return ""
return urllib.parse.quote(load.decode('utf-8', 'ignore'))
def set_load(self, packet, field, value):
"""
Helper method to retrieve load, as scapy doesn't recognize 'load' as
a field properly.
"""
if packet.haslayer("IP"):
del packet["IP"].len
value = urllib.parse.unquote(value)
value = value.encode('utf-8')
self.layer.payload = Raw(value)
def gen_load(self, field):
"""
Helper method to generate a random load, as scapy doesn't recognize 'load'
as a field properly.
"""
load = ''.join([random.choice(string.ascii_lowercase + string.digits) for k in range(10)])
return urllib.parse.quote(load)
class IPLayer(Layer):
"""
Defines an interface to access IP header fields.
"""
name = "IP"
protocol = IP
_fields = [
'version',
'ihl',
'tos',
'len',
'id',
'flags',
'frag',
'ttl',
'proto',
'chksum',
'src',
'dst',
'load'
]
fields = _fields
def __init__(self, layer):
"""
Initializes the IP layer.
"""
Layer.__init__(self, layer)
self.getters = {
"flags" : self.get_flags,
"load" : self.get_load
}
self.setters = {
"flags" : self.set_flags,
"load" : self.set_load
}
self.generators = {
"src" : self.gen_ip,
"dst" : self.gen_ip,
"chksum" : self.gen_chksum,
"len" : self.gen_len,
"load" : self.gen_load,
"flags" : self.gen_flags
}
def gen_len(self, field):
"""
Generates a valid IP length. Scapy breaks if the length is set to 0, so
return a random int starting at 1.
"""
return random.randint(1, 500)
def gen_chksum(self, field):
"""
Generates a checksum.
"""
return random.randint(1, 65535)
def gen_ip(self, field):
"""
Generates an IP address.
"""
return RandIP()._fix()
def get_flags(self, field):
"""
Retrieves flags as a string.
"""
return str(self.layer.flags)
def set_flags(self, packet, field, value):
"""
Sets the flags field. There is a bug in scapy, if you retrieve an empty
flags field, it will return "", but you cannot set this value back.
To reproduce this bug:
>>> setattr(IP(), "flags", str(IP().flags)) # raises a ValueError
To handle this case, this method converts empty string to zero so that
it can be safely stored.
"""
if value == "":
value = 0
self.layer.flags = value
def gen_flags(self, field):
"""
Generates random valid flags.
"""
sample = fuzz(self.protocol())
# Since scapy lazily evaluates fuzzing, we first must set a
# legitimate value for scapy to evaluate what combination of flags it is
sample.flags = sample.flags
return str(sample.flags)
class TCPLayer(Layer):
"""
Defines an interface to access TCP header fields.
"""
name = "TCP"
protocol = TCP
_fields = [
'sport',
'dport',
'seq',
'ack',
'dataofs',
'reserved',
'flags',
'window',
'chksum',
'urgptr',
'load',
'options-eol',
'options-nop',
'options-mss',
'options-wscale',
'options-sackok',
'options-sack',
'options-timestamp',
'options-altchksum',
'options-altchksumopt',
'options-md5header',
'options-uto'
]
fields = _fields
options_names = {
"eol": 0,
"nop": 1,
"mss": 2,
"wscale": 3,
"sackok": 4,
"sack": 5,
#"echo" : 6,
#"echo_reply" : 7,
"timestamp": 8,
"altchksum": 14,
"altchksumopt": 15,
"md5header": 19,
#"quick_start" : 27,
"uto": 28
#"authentication": 29,
#"experiment": 254
}
# Each entry is Kind: length
options_length = {
0: 0, # EOL
1: 0, # NOP
2: 2, # MSS
3: 1, # WScale
4: 0, # SAckOK
5: 0, # SAck
6: 4, # Echo
7: 4, # Echo Reply
8: 8, # Timestamp
14: 3, # AltChkSum
15: 0, # AltChkSumOpt
19: 16, # MD5header Option
27: 6, # Quick-Start response
28: 2, # User Timeout Option
29: 4, # TCP Authentication Option
254: 8, # Experiment
}
# Required by scapy
scapy_options = {
0: "EOL",
1: "NOP",
2: "MSS",
3: "WScale",
4: "SAckOK",
5: "SAck",
8: "Timestamp",
14: "AltChkSum",
15: "AltChkSumOpt",
28: "UTO",
# 254:"Experiment" # scapy has two versions of this, so it doesn't work
}
def __init__(self, layer):
"""
Initializes the TCP layer.
"""
Layer.__init__(self, layer)
# Special methods to help access fields that cannot be accessed normally
self.getters = {
'load' : self.get_load,
'options' : self.get_options
}
self.setters = {
'load' : self.set_load,
'options' : self.set_options
}
# Special methods to help generate fields that cannot be generated normally
self.generators = {
'load' : self.gen_load,
'dataofs' : self.gen_dataofs,
'flags' : self.gen_flags,
'chksum' : self.gen_chksum,
'options' : self.gen_options
}
def gen_chksum(self, field):
"""
Generates a checksum.
"""
return random.randint(1, 65535)
def gen_dataofs(self, field):
"""
Generates a valid value for the data offset field.
"""
# Dataofs is a 4 bit header, so a max of 15
return random.randint(1, 15)
def gen_flags(self, field):
"""
Generates a random set of flags. 50% of the time it picks randomly from
a list of real flags, otherwise it returns fuzzed flags.
"""
if random.random() < 0.5:
return random.choice(['S', 'A', 'SA', 'PA', 'FA', 'R', 'P', 'F', 'RA', ''])
else:
sample = fuzz(self.protocol())
# Since scapy lazily evaluates fuzzing, we first must set a
# legitimate value for scapy to evaluate what combination of flags it is
sample.flags = sample.flags
return str(sample.flags)
def get_options(self, field):
"""
Helper method to retrieve options.
"""
base, req_option = field.split("-")
assert base == "options", "get_options can only be used to fetch options."
option_type = self.option_str_to_int(req_option)
i = 0
# First, check if the option is already present in the packet
for option in self.layer.options:
# Scapy may try to be helpful and return the string of the option
next_option = self.option_str_to_int(option[0])
if option_type == next_option:
_name, value = self.layer.options[i]
# Some options (timestamp, checksums, nop) store their value in a
# tuple.
if isinstance(value, tuple):
# Scapy returns values in any of these types
if value in [None, b'', ()]:
return ''
value = value[0]
if value in [None, b'', ()]:
return ''
if req_option == "md5header":
return binascii.hexlify(value).decode("utf-8")
return value
i += 1
return ''
def set_options(self, packet, field, value):
"""
Helper method to set options.
"""
base, option = field.split("-")
assert base == "options", "Must use an options field with set_options"
option_type = self.option_str_to_int(option)
if type(value) == str:
# Prepare the value for storage in the packet
value = binascii.unhexlify(value)
# Scapy requires these options to be a tuple - since evaling this
# is not yet supported, for now, SAck will always be an empty tuple
if option in ["sack"]:
value = ()
# These options must be set as integers - if they didn't exist, they can
# be added like this
if option in ["timestamp", "mss", "wscale", "altchksum", "uto"] and not value:
value = 0
i = 0
# First, check if the option is already present in the packet
for option in self.layer.options:
# Scapy may try to be helpful and return the string of the option
next_option = self.option_str_to_int(option[0])
if option_type == next_option:
packet["TCP"].options[i] = self.format_option(option_type, value)
break
i += 1
# If we didn't break, the option doesn't exist in the packet currently.
else:
old_options_array = packet["TCP"].options
old_options_array.append(self.format_option(option_type, value))
packet["TCP"].options = old_options_array
# Let scapy recalculate the required values
del self.layer.chksum
del self.layer.dataofs
if packet.haslayer("IP"):
del packet["IP"].chksum
del packet["IP"].len
return True
def gen_options(self, field):
"""
Helper method to set options.
"""
_, option = field.split("-")
option_num = self.options_names[option]
length = self.options_length[option_num]
data = b''
if length > 0:
data = os.urandom(length)
data = binascii.hexlify(data).decode()
# MSS must be a 2-byte int
if option_num == 2:
data = random.randint(0, 65535)
# WScale must be a 1-byte int
elif option_num == 3:
data = random.randint(0, 255)
# Timestamp must be an int
elif option_num == 8:
data = random.randint(0, 4294967294)
elif option_num == 14:
data = random.randint(0, 255)
elif option_num == 28:
data = random.randint(0, 255)
return data
def option_str_to_int(self, option):
"""
Takes a string representation of an option and returns the option integer code.
"""
if type(option) == int:
return option
assert "-" not in option, "Must be given specific option: %s." % option
for val in self.scapy_options:
if self.scapy_options[val].lower() == option.lower():
return val
if " " in option:
option = option.replace(" ", "_").lower()
if option.lower() in self.options_names:
return self.options_names[option.lower()]
def format_option(self, options_int, value):
"""
Formats the options so they will work with scapy.
"""
# NOPs
if options_int == 1:
return (self.scapy_options[options_int], ())
elif options_int in [5]:
return (self.scapy_options[options_int], value)
# Timestamp
elif options_int in [8, 14]:
return (self.scapy_options[options_int], (value, 0))
elif options_int in self.scapy_options:
return (self.scapy_options[options_int], value)
else:
return (options_int, value)
class UDPLayer(Layer):
"""
Defines an interface to access UDP header fields.
"""
name = "UDP"
protocol = UDP
_fields = [
"sport",
"dport",
"chksum",
"len",
"load"
]
fields = _fields
def __init__(self, layer):
"""
Initializes the UDP layer.
"""
Layer.__init__(self, layer)
self.getters = {
'load' : self.get_load,
}
self.setters = {
'load' : self.set_load,
}
self.generators = {
'load' : self.gen_load,
}

240
actions/packet.py
Unescape Escape View File

@ -0,0 +1,240 @@
import copy
import random
import actions.layer
_SUPPORTED_LAYERS = [
actions.layer.IPLayer,
actions.layer.TCPLayer,
actions.layer.UDPLayer
]
SUPPORTED_LAYERS = _SUPPORTED_LAYERS
class Packet():
"""
Defines a Packet class, a convenience wrapper around
scapy packets for ease of use.
"""
def __init__(self, packet=None):
"""
Initializes the packet object.
"""
self.packet = packet
self.layers = self.setup_layers()
self.sleep = 0
def __str__(self):
"""
Defines string representation for the packet.
"""
return self._str_packet(self.packet)
@staticmethod
def _str_packet(packet):
"""
Static method to print a scapy packet.
"""
if packet.haslayer("TCP"):
return "TCP %s:%d --> %s:%d [%s] %s: %s" % (
packet["IP"].src,
packet["TCP"].sport,
packet["IP"].dst,
packet["TCP"].dport,
packet["TCP"].sprintf('%TCP.flags%'),
str(packet["TCP"].chksum),
Packet._str_load(packet["TCP"], "TCP"))
elif packet.haslayer("UDP"):
return "UDP %s:%d --> %s:%d %s: %s" % (
packet["IP"].src,
packet["UDP"].sport,
packet["IP"].dst,
packet["UDP"].dport,
str(packet["UDP"].chksum),
Packet._str_load(packet["UDP"], "UDP"))
load = ""
if hasattr(packet["IP"], "load"):
load = str(bytes(packet["IP"].load))
return "%s --> %s: %s" % (
packet["IP"].src,
packet["IP"].dst,
load)
@staticmethod
def _str_load(packet, protocol):
"""
Prints packet payload
"""
return str(packet[protocol].payload)
def __bytes__(self):
"""
Returns packet's binary representation.
"""
return bytes(self.packet)
def show(self, **kwargs):
"""
Calls scapy's show method.
"""
return self.packet.show(**kwargs)
def show2(self, **kwargs):
"""
Calls scapy's show method.
"""
return self.packet.show2(**kwargs)
def read_layers(self):
"""
Generator that yields parsed Layer objects from the protocols in the given packet.
"""
iter_packet = self.packet
while iter_packet:
if iter_packet.name.lower() == "raw":
return
parsed_layer = Packet.parse_layer(iter_packet)
if parsed_layer:
yield parsed_layer
iter_packet = parsed_layer.get_next_layer()
else:
iter_packet = iter_packet.payload
def has_supported_layers(self):
"""
Checks if this packet contains supported layers.
"""
return bool(self.layers)
def setup_layers(self):
"""
Sets up a lookup dictionary for the given layers in this packet.
"""
layers = {}
for layer in self.read_layers():
layers[layer.name.upper()] = layer
return layers
def copy(self):
"""
Deep copies this packet. This method is required because it is not safe
to use copy.deepcopy on this entire packet object, because the parsed layers
become disassociated with the underlying packet layers, which breaks layer
setting.
"""
return Packet(copy.deepcopy(self.packet))
@staticmethod
def parse_layer(to_parse):
"""
Takes a given scapy layer object and returns a Geneva Layer object.
"""
for layer in SUPPORTED_LAYERS:
if layer.name_matches(to_parse.name):
return layer(to_parse)
def haslayer(self, layer):
"""
Checks if a given layer is in the packet.
"""
return self.packet.haslayer(layer)
def __getitem__(self, item):
"""
Returns a layer.
"""
return self.packet[item]
def set(self, str_protocol, field, value):
"""
Sets the given protocol field to the given value.
Raises AssertionError if the protocol is not present.
"""
assert self.haslayer(str_protocol), "Given protocol %s is not in packet." % str_protocol
assert str_protocol in self.layers, "Given protocol %s is not permitted." % str_protocol
# Recalculate the checksums
if self.haslayer("IP"):
del self.packet["IP"].chksum
if self.haslayer("TCP"):
del self.packet["TCP"].chksum
return self.layers[str_protocol].set(self.packet, field, value)
def get(self, str_protocol, field):
"""
Retrieves the value of a given field for a given protocol.
Raises AssertionError if the protocol is not present.
"""
assert self.haslayer(str_protocol), "Given protocol %s is not in packet." % str_protocol
assert str_protocol in self.layers, "Given protocol %s is not permitted." % str_protocol
return self.layers[str_protocol].get(field)
def gen(self, str_protocol, field):
"""
Generates a value of a given field for a given protocol.
Raises AssertionError if the protocol is not present.
"""
assert self.haslayer(str_protocol), "Given protocol %s is not in packet." % str_protocol
assert str_protocol in self.layers, "Given protocol %s is not permitted." % str_protocol
return self.layers[str_protocol].gen(field)
@classmethod
def parse(cls, str_protocol, field, value):
"""
Parses a given value for a given field of a given protocool.
Raises AssertionError if the protocol is not present.
"""
parsing_layer = None
for layer in SUPPORTED_LAYERS:
if layer.name_matches(str_protocol):
parsing_layer = layer(None)
assert parsing_layer, "Given protocol %s is not permitted." % str_protocol
return parsing_layer.parse(field, value)
def get_random_layer(self):
"""
Retrieves a random layer from this packet.
"""
return self.layers[random.choice(list(self.layers.keys()))]
def get_random(self):
"""
Retrieves a random protocol, field, and value from this packet.
"""
layer = self.get_random_layer()
field, value = layer.get_random()
return layer.protocol, field, value
@staticmethod
def gen_random():
"""
Generates a possible random protocol, field, and value.
"""
# layer is a Geneva Layer class - to instantiate it, we must give it a layer
# to use. Every Geneva Layer stores the underlying scapy layer it wraps,
# so simply invoke that as a default.
layer = random.choice(SUPPORTED_LAYERS)
layer_obj = layer(layer.protocol())
field, value = layer_obj.gen_random()
return layer.protocol, field, value
@staticmethod
def get_supported_protocol(protocol):
"""
Checks if the given protocol exists in the SUPPORTED_LAYERS list.
"""
for layer in SUPPORTED_LAYERS:
if layer.name_matches(protocol.upper()):
return layer
return None

37
actions/sleep.py
Unescape Escape View File

@ -0,0 +1,37 @@
from actions.action import Action
class SleepAction(Action):
def __init__(self, time=1, environment_id=None):
Action.__init__(self, "sleep", "out")
self.terminal = False
self.branching = False
self.time = time
def run(self, packet, logger):
"""
The sleep action simply passes along the packet it was given with an instruction for how long the engine should sleep before sending it.
"""
logger.debug(" - Adding %d sleep to given packet." % self.time)
packet.sleep = self.time
return packet, None
def __str__(self):
"""
Returns a string representation.
"""
s = Action.__str__(self)
s += "{%d}" % self.time
return s
def parse(self, string, logger):
"""
Parses a string representation for this object.
"""
try:
if string:
self.time = float(string)
except ValueError:
logger.exception("Cannot parse time %s" % string)
return False
return True

85
actions/sniffer.py
Unescape Escape View File

@ -0,0 +1,85 @@
import threading
import os
import actions.packet
from scapy.all import sniff
from scapy.utils import PcapWriter
class Sniffer():
"""
The sniffer class lets the user begin and end sniffing whenever in a given location with a port to filter on.
Call start_sniffing to begin sniffing and stop_sniffing to stop sniffing.
"""
def __init__(self, location, port, logger):
"""
Intializes a sniffer object.
Needs a location and a port to filter on.
"""
self.stop_sniffing_flag = False
self.location = location
self.port = port
self.pcap_thread = None
self.packet_dumper = None
self.logger = logger
full_path = os.path.dirname(location)
assert port, "Need to specify a port in order to launch a sniffer"
if not os.path.exists(full_path):
os.makedirs(full_path)
def __packet_callback(self, scapy_packet):
"""
This callback is called whenever a packet is applied.
Returns true if it should finish, otherwise, returns false.
"""
packet = actions.packet.Packet(scapy_packet)
for proto in ["TCP", "UDP"]:
if(packet.haslayer(proto) and ((packet[proto].sport == self.port) or (packet[proto].dport == self.port))):
break
else:
return self.stop_sniffing_flag
self.logger.debug(str(packet))
self.packet_dumper.write(scapy_packet)
return self.stop_sniffing_flag
def __spawn_sniffer(self):
"""
Saves pcaps to a file. Should be run as a thread.
Ends when the stop_sniffing_flag is set. Should not be called by user
"""
self.packet_dumper = PcapWriter(self.location, append=True, sync=True)
while(self.stop_sniffing_flag == False):
sniff(stop_filter=self.__packet_callback, timeout=1)
def start_sniffing(self):
"""
Starts sniffing. Should be called by user.
"""
self.stop_sniffing_flag = False
self.pcap_thread = threading.Thread(target=self.__spawn_sniffer)
self.pcap_thread.start()
self.logger.debug("Sniffer starting to port %d" % self.port)
def __enter__(self):
"""
Defines a context manager for this sniffer; simply starts sniffing.
"""
self.start_sniffing()
return self
def __exit__(self, exc_type, exc_value, tb):
"""
Defines exit context manager behavior for this sniffer; simply stops sniffing.
"""
self.stop_sniffing()
def stop_sniffing(self):
"""
Stops the sniffer by setting the flag and calling join
"""
if(self.pcap_thread):
self.stop_sniffing_flag = True
self.pcap_thread.join()
self.logger.debug("Sniffer stopping")

88
actions/strategy.py
Unescape Escape View File

@ -0,0 +1,88 @@
import random
import actions.utils
import actions.tree
class Strategy(object):
def __init__(self, in_actions, out_actions, environment_id=None):
self.in_actions = in_actions
self.out_actions = out_actions
self.in_enabled = True
self.out_enabled = True
self.environment_id = environment_id
self.fitness = -1000
def __str__(self):
"""
Builds a string describing the action trees for this strategy.
"""
return "%s \/ %s" % (self.str_forest(self.out_actions).strip(), self.str_forest(self.in_actions).strip())
def __len__(self):
"""
Returns the number of actions in this strategy.