Controlled Vocabularies

docs/examples/index.rst

@@ -0,0 +1,195 @@
+Examples
+========
+
+This page includes some basic examples of creating and parsing STIX content.
+
+There are a couple things we do in these examples for purposes of demonstration
+that shouldn't be done in production code:
+
+* When calling ``to_xml()``, we use ``include_namespaces=False``. This is to
+  make the example output easier to read, but means the resulting output
+  cannot be successfully parsed. The XML parser doesn't know what namespaces
+  to use if they aren't included. In production code, you should explicitly
+  set ``include_namespaces`` to ``True`` or omit it entirely (``True`` is the
+  default).
+
+* In some examples, we use ``set_id_method(IDGenerator.METHOD_INT)`` to make
+  IDs for STIX constructs easier to read and cross-reference within the XML
+  document. In production code, you should omit this statement, which causes
+  random UUIDs to be created instead, or create explicit IDs yourself for STIX
+  constructs.
+
+See the `STIX Idioms <http://stixproject.github.io/documentation/idioms/>`_
+documentation for more great examples of how to use ``python-stix``.
+
+
+Creating a STIX Package
+-----------------------
+.. testcode::
+
+    from stix.core import STIXPackage, STIXHeader
+    from stix.utils import IDGenerator, set_id_method
+
+    set_id_method(IDGenerator.METHOD_INT) # For testing and demonstration only!
+
+    stix_package = STIXPackage()
+    stix_header = STIXHeader()
+    stix_header.description = "Getting Started!"
+    stix_package.stix_header = stix_header
+
+    print stix_package.to_xml(include_namespaces=False)
+
+Which outputs:
+
+.. testoutput::
+
+    <stix:STIX_Package id="example:Package-1" version="1.1.1" timestamp="2014-08-12T18:03:44.240457+00:00">
+        <stix:STIX_Header>
+            <stix:Description>Getting Started!</stix:Description>
+        </stix:STIX_Header>
+    </stix:STIX_Package>
+
+
+ID Namespaces
+-------------
+By default, ``python-stix`` sets the default ID namespace to
+``http://example.com`` with an alias of ``example``. This results in STIX
+id declarations that look like
+``id="example:Package-2813128d-f45e-41f7-b10a-20a5656e3785"``.
+
+To change this, use the ``stix.utils.set_id_namespace()`` method which takes
+a dictionary as a parameter.
+
+.. testcode::
+
+    from stix.core import STIXPackage
+    from stix.utils import set_id_namespace
+
+    NAMESPACE = {"http://MY-NAMESPACE.com" : "myNS"}
+    set_id_namespace(NAMESPACE) # new ids will be prefixed by "myNS"
+
+    stix_package = STIXPackage() # id will be created automatically
+    print stix_package.to_xml()
+
+Which outputs:
+
+.. testoutput::
+
+    <stix:STIX_Package
+        xmlns:myNS="http://MY-NAMESPACE.com"
+        xmlns:stixCommon="http://stix.mitre.org/common-1"
+        xmlns:stixVocabs="http://stix.mitre.org/default_vocabularies-1"
+        xmlns:stix="http://stix.mitre.org/stix-1"
+        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+        xsi:schemaLocation="
+        http://stix.mitre.org/common-1 http://stix.mitre.org/XMLSchema/common/1.1.1/stix_common.xsd
+        http://stix.mitre.org/default_vocabularies-1 http://stix.mitre.org/XMLSchema/default_vocabularies/1.1.1/stix_default_vocabularies.xsd
+        http://stix.mitre.org/stix-1 http://stix.mitre.org/XMLSchema/core/1.1.1/stix_core.xsd"
+        id="myNS:Package-b2039368-9476-4a5b-8c1d-0ef5d1b37e06" version="1.1.1" timestamp="2014-08-12T18:15:33.603457+00:00"/>
+
+Success! The ``xmlns:myNS="http://MY-NAMESPACE.com"`` matches our ``NAMESPACE``
+dictionary and the ``id`` attribute includes the ``myNS`` namespace alias.
+
+Controlled Vocabularies: VocabString
+------------------------------------
+Many fields in STIX leverage the ``stixCommon:ControlledVocabularyStringType``,
+which acts as a base type for controlled vocabulary implementations. The STIX
+language defines a set of default controlled vocabularies which are  found in
+the ``stix_default_vocabs.xsd`` XML Schema file.
+
+The ``python-stix`` library contains a ``stix.common.vocabs`` module, which
+defines the ``VocabString`` class implementation of the schema
+``ControlledVocabularyStringType`` as well as ``VocabString`` implementations
+which correspond to default controlled vocabularies.
+
+For example, the ``stix_default_vocabularies.xsd`` schema defines a controlled
+vocabulary for STIX Package Intents: ``PackageIntentVocab-1.0``. The
+``stix.common.vocabs`` module contains an analogous ``PackageIntent`` class,
+which acts as a derivation of ``VocabString``.
+
+Each ``VocabString`` implementation contains:
+
+* A static list of class-level term attributes, each beginning with ``TERM_`
+  (e.g., ``TERM_INDICATORS``)
+
+* A tuple containing all allowed vocabulary terms: ``ALLOWED_VALUES``, which is
+  use for input validation
+
+* Methods found on ``stix.Entity``, such as ``to_xml()``, ``to_dict()``,
+  ``from_dict()``, etc.
+
+
+Interacting With VocabString Fields
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+The following sections define ways of interacting with VocabString fields.
+
+Default Vocabulary Terms
+########################
+The STIX Language often suggested a default controlled vocabulary type for a
+given controlled vocabulary field. Each controlled vocabulary contains an
+enumeration of allowed terms.
+
+Each ``VocabString`` implementation found in the ``stix.common.vocabs`` module
+contains static class-level attributes for each vocabulary term. When setting
+controlled vocabulary field values, it is recommended that users take advantage
+of these class-level attributes.
+
+The following demonstrates setting the ``Package_Intent`` field with a default
+vocabulary term. Note that the ``STIXHeader.package_intents`` property returns
+a list. As such, we use the ``append()`` method to add terms. Other STIX
+controlled vocabulary fields may only allow one value rather than a list of
+values.
+
+.. testcode::
+
+    from stix.core import STIXHeader
+    from stix.common.vocabs import PackageIntent
+
+    header = STIXHeader()
+    header.package_intents.append(PackageIntent.TERM_INDICATORS)
+
+    print header.to_xml(include_namespaces=False)
+
+Which outputs:
+
+.. testoutput::
+
+    <stix:STIXHeaderType>
+        <stix:Package_Intent xsi:type="stixVocabs:PackageIntentVocab-1.0">Indicators</stix:Package_Intent>
+    </stix:STIXHeaderType>
+
+Non-Default Vocabulary Terms
+############################
+Though it is suggested, STIX content authors are not required to use the default
+controlled vocabulary for a given field. As such, ``python-stix`` allows users
+to pass in non-default values for controlled vocabulary fields.
+
+To set a controlled vocabulary to a non-default vocabulary term, pass a
+``VocabString`` instance into a controlled vocabulary field.
+
+A raw ``VocabString`` field will contain no ``xsi:type`` information or
+``ALLOWED_VALUES`` members, which removes the input and schema validation
+requirements.
+
+.. testcode::
+
+    from stix.core import STIXHeader
+    from stix.common.vocabs import VocabString, PackageIntent
+
+    header = STIXHeader()
+    non_default_term = VocabString("NON-DEFAULT VOCABULARY TERM")
+    header.package_intents.append(non_default_term)
+
+    print header.to_xml(include_namespaces=False)
+
+Which outputs:
+
+.. testoutput::
+
+    <stix:STIXHeaderType>
+        <stix:Package_Intent>NON-DEFAULT VOCABULARY TERM</stix:Package_Intent>
+    </stix:STIXHeaderType>
+
+Notice that the ``<stix:Package_Intent>`` field does not have an ``xsi:type``
+attribute. As such, this field can contain any string value and is not bound
+by a controlled vocabulary enumeration of terms.

docs/index.rst

@@ -43,6 +43,7 @@ Contents
 
    installation
    getting_started
+   examples/index
    api_vs_bindings/index
 
 API Reference

examples/vocabstrings.py

@@ -0,0 +1,53 @@
+#!/usr/bin/env python
+# Copyright (c) 2014, The MITRE Corporation. All rights reserved.
+# See LICENSE.txt for complete terms.
+
+'''
+File: vocabstrings.
+
+Demonstrates uses of VocabString for STIX Controlled Vocabularies
+'''
+import sys
+from stix.common.vocabs import VocabString
+
+def main():
+    # Adding a Package_Intent field to a STIX_Header.
+    from stix.core import STIXHeader
+    from stix.common.vocabs import PackageIntent
+
+    # Create STIXHeader instance
+    header = STIXHeader()
+    header.package_intents.append(PackageIntent.TERM_INDICATORS)
+
+    # To add a Package_Intent value that exists outside of the
+    # PackageIntentVocab controlled vocabulary, we pass in an
+    # instance of VocabString.
+    #
+    # This will create a new Package_Intent field without an
+    # xsi:type and will not perform any validation of input terms.
+    #
+    # Passing in an instance of VocabString works for every
+    # ControlledVocabularyStringType field (or in python-stix,
+    # every VocabString field).
+
+    non_default_value = VocabString("NON STANDARD PACKAGE INTENT")
+    header.package_intents.append(non_default_value)
+
+    print header.to_xml()
+
+    # NOTE: Passing in a str value that is not included in the list
+    # of default CV terms will raise a ValueError. This is why we pass
+    # in a VocabString instance.
+    #
+    # Example:
+    try:
+        print("[-] Attempting to add an str instance that does not exist "
+              "in the PackageIntent ALLOWED_VALUES list")
+        header.package_intents.append("THIS WILL THROW A VALUEERROR")
+    except Exception as ex:
+        print "[!] As expected, that failed. Here's the Exception message:"
+        print "[!]", str(ex)
+
+if __name__ == '__main__':
+    main()
+

stix/base.py

@@ -22,35 +22,42 @@ def from_obj(cls, obj):
         """Create an object from a binding object"""
         raise NotImplementedError()
 
-    def _get_namespaces(self, ns_dict):
+    def _get_namespaces(self, ns_dict=None):
         import stix.utils.nsparser as nsparser
-        namespace_parser = nsparser.NamespaceParser()
-        all_ns_dict = namespace_parser.get_namespaces(self, ns_dict=ns_dict)        
+        parser = nsparser.NamespaceParser()
+        all_ns_dict = parser.get_namespaces(self, ns_dict=ns_dict)
         return all_ns_dict
 
-    def _get_schema_locations(self, ns_dict=None):
+    def _get_schema_locations(self, ns_dict=None, schemaloc_dict=None):
         import stix.utils.nsparser as nsparser
-        schemaloc_dict = nsparser.NamespaceParser().get_namespace_schemalocation_dict(self, ns_dict=ns_dict)
-        return schemaloc_dict
+        parser = nsparser.NamespaceParser()
+        all_schemaloc_dict = \
+            parser.get_namespace_schemalocation_dict(self, ns_dict=ns_dict, schemaloc_dict=schemaloc_dict)
+        return all_schemaloc_dict
 
-    def to_xml(self, include_namespaces=True, ns_dict=None, pretty=True):
+    def to_xml(self, include_namespaces=True, ns_dict=None, schemaloc_dict=None, pretty=True):
         """Export an object as an XML String""" 
         s = StringIO()
         namespace_def = ""
 
         import stix.utils.nsparser as nsparser
         if include_namespaces:
-            if not ns_dict: ns_dict = {}
             all_ns_dict = self._get_namespaces(ns_dict)
-            schemaloc_dict = self._get_schema_locations(all_ns_dict)
-            namespace_def = nsparser.NamespaceParser().get_namespace_def_str(all_ns_dict, schemaloc_dict)
+            all_schemaloc_dict = \
+                self._get_schema_locations(all_ns_dict, schemaloc_dict)
+
+            parser = nsparser.NamespaceParser()
+            namespace_def = parser.get_namespace_def_str(all_ns_dict,
+                                                         all_schemaloc_dict)
         else:
-            all_ns_dict = dict(nsparser.DEFAULT_STIX_NS_TO_PREFIX.items() + nsparser.DEFAULT_EXT_TO_PREFIX.items())
+            all_ns_dict = dict(nsparser.DEFAULT_STIX_NS_TO_PREFIX.items() +
+                               nsparser.DEFAULT_EXT_TO_PREFIX.items())
 
         if not pretty:
             namespace_def = namespace_def.replace('\n\t', ' ')
 
-        self.to_obj().export(s, 0, all_ns_dict, pretty_print=pretty, namespacedef_=namespace_def)
+        self.to_obj().export(s, 0, all_ns_dict, pretty_print=pretty,
+                             namespacedef_=namespace_def)
         return s.getvalue()
 
     def _get_children(self):
@@ -62,7 +69,9 @@ def _get_children(self):
                 yield obj
             elif isinstance(obj, (tuple, collections.MutableSequence)):
                 for item in obj:
-                    if isinstance(item, Entity) or isinstance(item, Observable) or isinstance(item, cybox.Entity):
+                    if (isinstance(item, Entity) or
+                        isinstance(item, Observable) or
+                        isinstance(item, cybox.Entity)):
                         yield item
 
     def to_json(self):

stix/common/information_source.py

@@ -135,7 +135,7 @@ def roles(self):
 
     @roles.setter
     def roles(self, value):
-        self._roles = []
+        self._roles = Roles()
 
         if not value:
             return
@@ -243,3 +243,10 @@ class ContributingSources(stix.EntityList):
     _contained_type = InformationSource
     _inner_name = "sources"
 
+
+class Roles(stix.EntityList):
+    _namespace = "http://stix.mitre.org/common-1"
+    _contained_type = VocabString
+
+    def _fix_value(self, value):
+        return InformationSourceRole(value)

stix/common/vocabs.py

@@ -67,7 +67,7 @@ def lookup_class(xsi_type):
             if xsi_type in k:
                 return v
 
-        raise ValueError("Unregistered xsi:type %s" % xsi_type)
+        return VocabString
 
     def to_obj(self, return_obj=None):
         if not return_obj:

stix/core/stix_header.py

@@ -62,7 +62,7 @@ def package_intents(self):
 
     @package_intents.setter
     def package_intents(self, value):
-        self._package_intents = []
+        self._package_intents = PackageIntents()
         if not value:
             return
         elif isinstance(value, list):
@@ -168,3 +168,11 @@ def to_dict(self):
             d['profiles'] = self.profiles
 
         return d
+
+
+class PackageIntents(stix.EntityList):
+    _namespace = "http://stix.mitre.org/stix-1"
+    _contained_type = VocabString
+
+    def _fix_value(self, value):
+        return PackageIntent(value)