This post is the third of a series on Threat Intelligence Automation topic.
Post 1: Architecture and Hardening of MineMeld
Post 2: Foundation: write a custom prototype and SOC integration
Post 4: Search received IoC events with Splunk
Post 5: Connect to a TAXII service
After building the architecture and integrating the InfoSec feeds from italian CERT-PA into MineMeld and the near-real-time SOC engine, it’s time to put another brick to build an effective community: export internal IoC to the community in a standard format so authorized parties can get it and use them as they want.
The ultimate goal is to build a community that can share IoC using a standard language and a transport mechanism (STIX/TAXXI) getting data from heterogeneous sources (more integration examples in next posts) and injecting data into the community network.
So let’s start with the configuration steps.
Following picture shows an example of a target community network where:
- there is a central hub that sends IoC to consumers and receives IoC from producers (spoke). This is typically done by a a CERT that – hopefully 🙂 – does analysis and contextualization before sending and after receiving IoC;
- each peripheral entity (spoke) can also build peer-to-peer connections with other parties to directly share specific IoC.
In this post I show you how to become a producer exporting URL, IP and domains in STIX/TAXII format; the reference network architecture is described in post 1.
From MineMeld CONFIG page, IMPORT the following config (remember to COMMIT at the end):
- press APPEND if you have a working config already in place, like the config made on post 2. Remember to do a BACKUP before, just in case;
- press REPLACE if your current config can be erased.
nodes: TI-url-export: inputs: [] output: true prototype: stdlib.listURLGeneric TI-domain-export: inputs: [] output: true prototype: stdlib.listDomainGeneric TI-ipv4-export: inputs: [] output: true prototype: stdlib.listIPv4Generic TAXII-output: inputs: - TI-url-export - TI-domain-export - TI-ipv4-export output: false prototype: stdlib.taxiiDataFeed
At the end you have
- TI-url-export
The miner where we add URL (uses stdlib.listURLGeneric prototype); - TI-domain-export
The miner where we add domains (uses stdlib.listDomainGeneric prototype); - TI-ipv4-export
The miner where we add IP addresses (uses stdlib.listIPv4Generic prototype); - TAXII-output
The output node that exports data received from the 3 miners in STIX/TAXII format (stdlib.taxiiDataFeed prototype). This is what I called the producer.
And this is the connection graph.
Now it’s time to fill the miners with some IoC.
This can be done from NODES page, clicking on each miner (TI-*-export), then on the INDICATORS icon (left) and then filling the fields (next image is for domain indicators).
After that the output node TAXII-output has the 3 indicators added.
The TAXII node is now ready to be queried from a TAXII client using the DISCOVERY SERVICE url: https://192.168.X.X/taxii-discovery-service
To test the TAXII output we can use Cabby that is a simple python TAXII client implemetation. Here the install guide, just install virtualenv before (on Ubuntu: $ sudo apt-get isntall virtualenv).
I tested some command line tools as follow:
// note: "--verify no" option added because in test lab I have a self-signed certificate // service discovery (venv) gmellini@SandTigerShark:~$ taxii-discovery --host 192.168.X.X --path /taxii-discovery-service --https --verify no 2017-08-30 16:37:49,265 INFO: Sending Discovery_Request to https://192.168.X.X/taxii-discovery-service 2017-08-30 16:37:49,269 WARNING: /home/gmellini/venv/local/lib/python2.7/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning) 2017-08-30 16:37:49,289 INFO: 3 services discovered === Service Instance === Service Type: DISCOVERY Service Version: urn:taxii.mitre.org:services:1.1 Protocol Binding: urn:taxii.mitre.org:protocol:http:1.0 Service Address: https://192.168.X.X/taxii-discovery-service Message Binding: urn:taxii.mitre.org:message:xml:1.1 Available: True Message: None === Service Instance === Service Type: COLLECTION_MANAGEMENT Service Version: urn:taxii.mitre.org:services:1.1 Protocol Binding: urn:taxii.mitre.org:protocol:http:1.0 Service Address: https://192.168.X.X/taxii-collection-management-service Message Binding: urn:taxii.mitre.org:message:xml:1.1 Available: True Message: None === Service Instance === Service Type: POLL Service Version: urn:taxii.mitre.org:services:1.1 Protocol Binding: urn:taxii.mitre.org:protocol:http:1.0 Service Address: https://192.168.X.X/taxii-poll-service Message Binding: urn:taxii.mitre.org:message:xml:1.1 Available: True Message: None // get collection list (venv) gmellini@SandTigerShark:~$ taxii-collections --path https://192.168.X.X/taxii-collection-management-service --verify no 2017-08-30 16:53:46,540 INFO: Sending Collection_Information_Request to https://192.168.X.X/taxii-collection-management-service 2017-08-30 16:53:46,544 WARNING: /home/gmellini/venv/local/lib/python2.7/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning) === Data Collection Information === Collection Name: TAXII-output Collection Type: DATA_FEED Available: True Collection Description: TAXII-output Data Feed Supported Content: urn:stix.mitre.org:xml:1.1.1 === Polling Service Instance === Poll Protocol: urn:taxii.mitre.org:protocol:http:1.0 Poll Address: https://192.168.X.X/taxii-poll-service Message Binding: urn:taxii.mitre.org:message:xml:1.1 ================================== // poll discovered collections (venv) gmellini@SandTigerShark:~$ taxii-poll --host 192.168.X.X --https --verify no --collection TAXII-output --discovery /taxii-discovery-service 2017-08-30 16:59:38,174 INFO: Polling using data binding: ALL 2017-08-30 16:59:38,175 INFO: Sending Discovery_Request to https://192.168.X.X/taxii-discovery-service 2017-08-30 16:59:38,179 WARNING: /home/gmellini/venv/local/lib/python2.7/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning) 2017-08-30 16:59:38,200 INFO: 3 services discovered 2017-08-30 16:59:38,200 INFO: Sending Poll_Request to https://192.168.X.X/taxii-poll-service 2017-08-30 16:59:38,203 WARNING: /home/gmellini/venv/local/lib/python2.7/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning) <stix:STIX_Package xmlns:DomainNameObj="http://cybox.mitre.org/objects#DomainNameObject-1" xmlns:cybox="http://cybox.mitre.org/cybox-2" xmlns:cyboxCommon="http://cybox.mitre.org/common-2" xmlns:cyboxVocabs="http://cybox.mitre.org/default_vocabularies-2" xmlns:example="http://example.com" xmlns:indicator="http://stix.mitre.org/Indicator-2" xmlns:minemeld="https://go.paloaltonetworks.com/minemeld" xmlns:stix="http://stix.mitre.org/stix-1" xmlns:stixCommon="http://stix.mitre.org/common-1" xmlns:stixVocabs="http://stix.mitre.org/default_vocabularies-1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:taxii="http://taxii.mitre.org/messages/taxii_xml_binding-1" xmlns:taxii_11="http://taxii.mitre.org/messages/taxii_xml_binding-1.1" xmlns:tdq="http://taxii.mitre.org/query/taxii_default_query-1" id="minemeld:indicator-d196ce3b-d27c-4176-a131-5111c7024411" version="1.1.1" timestamp="2017-08-30T13:53:38.193519+00:00"> <stix:Indicators> <stix:Indicator id="minemeld:indicator-9dcf3ce3-305b-46c0-93db-88f9d937c69f" timestamp="2017-08-30T13:53:38.193821+00:00" xsi:type="indicator:IndicatorType"> <indicator:Title>domain: wwwww.brokendomain.broken</indicator:Title> <indicator:Type xsi:type="stixVocabs:IndicatorTypeVocab-1.1">URL Watchlist</indicator:Type> <indicator:Description>domain indicator from TI-domain-export</indicator:Description> <indicator:Observable id="minemeld:observable-8d6a2557-0831-4950-9092-7473bcd05010"> <cybox:Title>FQDN: wwwww.brokendomain.broken</cybox:Title> <cybox:Object id="minemeld:DomainName-960ffcd7-e9d5-4b16-bf02-ecfa6379b5b4"> <cybox:Properties xsi:type="DomainNameObj:DomainNameObjectType" type="FQDN"> <DomainNameObj:Value>wwwww.brokendomain.broken</DomainNameObj:Value> </cybox:Properties> </cybox:Object> </indicator:Observable> <indicator:Confidence timestamp="2017-08-30T13:53:38.194039+00:00"> <stixCommon:Value xsi:type="stixVocabs:HighMediumLowVocab-1.0">High</stixCommon:Value> </indicator:Confidence> </stix:Indicator> </stix:Indicators> </stix:STIX_Package> <stix:STIX_Package xmlns:AddressObj="http://cybox.mitre.org/objects#AddressObject-2" xmlns:cybox="http://cybox.mitre.org/cybox-2" xmlns:cyboxCommon="http://cybox.mitre.org/common-2" xmlns:cyboxVocabs="http://cybox.mitre.org/default_vocabularies-2" xmlns:example="http://example.com" xmlns:indicator="http://stix.mitre.org/Indicator-2" xmlns:minemeld="https://go.paloaltonetworks.com/minemeld" xmlns:stix="http://stix.mitre.org/stix-1" xmlns:stixCommon="http://stix.mitre.org/common-1" xmlns:stixVocabs="http://stix.mitre.org/default_vocabularies-1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:taxii="http://taxii.mitre.org/messages/taxii_xml_binding-1" xmlns:taxii_11="http://taxii.mitre.org/messages/taxii_xml_binding-1.1" xmlns:tdq="http://taxii.mitre.org/query/taxii_default_query-1" id="minemeld:indicator-82cf103a-3bcf-4dc0-af60-c35c051b9ce8" version="1.1.1" timestamp="2017-08-30T13:54:28.029792+00:00"> <stix:Indicators> <stix:Indicator id="minemeld:indicator-05a1fdff-35b0-4c12-8a23-2e7e74d96cd1" timestamp="2017-08-30T13:54:28.030074+00:00" xsi:type="indicator:IndicatorType"> <indicator:Title>IPv4: 1.1.1.1/32</indicator:Title> <indicator:Type xsi:type="stixVocabs:IndicatorTypeVocab-1.1">IP Watchlist</indicator:Type> <indicator:Description>IPv4 indicator from TI-ipv4-export</indicator:Description> <indicator:Observable id="minemeld:observable-f77e3d4d-62f6-4134-84b4-f2519338f672"> <cybox:Title>IPv4: 1.1.1.1/32</cybox:Title> <cybox:Object id="minemeld:Address-dbda63c8-0b0e-4c83-abf8-8fd510c4fa6b"> <cybox:Properties xsi:type="AddressObj:AddressObjectType" category="ipv4-addr"> <AddressObj:Address_Value>1.1.1.1/32</AddressObj:Address_Value> </cybox:Properties> </cybox:Object> </indicator:Observable> <indicator:Confidence timestamp="2017-08-30T13:54:28.030253+00:00"> <stixCommon:Value xsi:type="stixVocabs:HighMediumLowVocab-1.0">High</stixCommon:Value> </indicator:Confidence> </stix:Indicator> </stix:Indicators> </stix:STIX_Package> <stix:STIX_Package xmlns:URIObj="http://cybox.mitre.org/objects#URIObject-2" xmlns:cybox="http://cybox.mitre.org/cybox-2" xmlns:cyboxCommon="http://cybox.mitre.org/common-2" xmlns:cyboxVocabs="http://cybox.mitre.org/default_vocabularies-2" xmlns:example="http://example.com" xmlns:indicator="http://stix.mitre.org/Indicator-2" xmlns:minemeld="https://go.paloaltonetworks.com/minemeld" xmlns:stix="http://stix.mitre.org/stix-1" xmlns:stixCommon="http://stix.mitre.org/common-1" xmlns:stixVocabs="http://stix.mitre.org/default_vocabularies-1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:taxii="http://taxii.mitre.org/messages/taxii_xml_binding-1" xmlns:taxii_11="http://taxii.mitre.org/messages/taxii_xml_binding-1.1" xmlns:tdq="http://taxii.mitre.org/query/taxii_default_query-1" id="minemeld:indicator-c58d05b6-55cc-4071-9446-9d7b38613bc6" version="1.1.1" timestamp="2017-08-30T13:56:16.463180+00:00"> <stix:Indicators> <stix:Indicator id="minemeld:indicator-6ba0e59f-97e5-467e-a5de-c42353a53e87" timestamp="2017-08-30T13:56:16.463434+00:00" xsi:type="indicator:IndicatorType"> <indicator:Title>URL: https://wwww.brokensite.com/compromised.php</indicator:Title> <indicator:Type xsi:type="stixVocabs:IndicatorTypeVocab-1.1">Domain Watchlist</indicator:Type> <indicator:Description>URL indicator from TI-url-export</indicator:Description> <indicator:Observable id="minemeld:observable-de9352e3-8b56-4ec4-ba79-508c14d702ab"> <cybox:Title>URL: https://wwww.brokensite.com/compromised.php</cybox:Title> <cybox:Object id="minemeld:URI-82dd5db9-51fe-4c7b-86ff-e9c74b022aea"> <cybox:Properties xsi:type="URIObj:URIObjectType" type="URL"> <URIObj:Value>https://wwww.brokensite.com/compromised.php</URIObj:Value> </cybox:Properties> </cybox:Object> </indicator:Observable> <indicator:Confidence timestamp="2017-08-30T13:56:16.463591+00:00"> <stixCommon:Value xsi:type="stixVocabs:HighMediumLowVocab-1.0">High</stixCommon:Value> </indicator:Confidence> </stix:Indicator> </stix:Indicators> </stix:STIX_Package>
Cabby gets the 3 indicators (STIX) from TAXII-output collection. It works 🙂
The TAXII output is now available to anyone that knows the discovery URL.
Of course we can restrict the access only to authorized users in MineMeld.
To do this first create a new feed user from ADMIN – FEED USERS page (guest/guest) then an access tag for the user (public_taxii). This is the tag to be “attached” to the output feed to set authentication.
There is a warning; feed authentication must be explicitly enabled with following command (to see the new config just refresh the web interface with CTRL+F5).
$ sudo -u minemeld sh -c 'echo "FEEDS_AUTH_ENABLED: True" > /opt/minemeld/local/config/api/30-feeds-auth.yml'
Now from CONFIG page click on TAXII-output node and set public_taxii tag.
The non-authenticated Cabby service discovery command return an authorization error.
(venv) gmellini@SandTigerShark:~$ taxii-poll --host 192.168.X.X --https --verify no --collection TAXII-output --discovery /taxii-discovery-service
2017-08-30 17:47:36,842 INFO: Polling using data binding: ALL
2017-08-30 17:47:36,843 INFO: Sending Discovery_Request to https://192.168.X.X/taxii-discovery-service
2017-08-30 17:47:36,847 WARNING: /home/gmellini/venv/local/lib/python2.7/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
2017-08-30 17:47:36,866 ERROR: Can not autodiscover advertised services
2017-08-30 17:47:36,866 ERROR: HTTP Error: status code 401
(venv) gmellini@SandTigerShark:~$
Trying with guest credential works fine.
(venv) gmellini@SandTigerShark:~$ taxii-poll --host 192.168.X.X --https --verify no --collection TAXII-output --discovery /taxii-discovery-service --username guest --password guest
2017-08-30 17:49:59,900 INFO: Polling using data binding: ALL
2017-08-30 17:49:59,901 INFO: Sending Discovery_Request to https://192.168.X.X/taxii-discovery-service
2017-08-30 17:49:59,905 WARNING: /home/gmellini/venv/local/lib/python2.7/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
2017-08-30 17:49:59,925 INFO: 3 services discovered
2017-08-30 17:49:59,925 INFO: Sending Poll_Request to https://192.168.X.X/taxii-poll-service
2017-08-30 17:49:59,929 WARNING: /home/gmellini/venv/local/lib/python2.7/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
<stix:STIX_Package [...]
Last but not least when you enable feed authentication all your existing non-authenticated feeds stop working; you need to tell MineMeld that these feeds are anonymous feed adding anonymous tag of course; this is a default tag (you can see in the tag list).
Do this if you have an existing configuration or if you have in place the config from post 2; in this case 4SPLUNK output feed must look like this.
Enjoy!
Scubarda 
4 thoughts on “MineMeld: threat intelligence automation – export internal IoC to the community [3]”