ibm-research/SI-Cyber

The SI-Cyber dataset is derived from subgraphs extracted from the MITRE D3FEND ontology and contains a total of 5,934 graph edges.

The dataset is organized into 7,980 query clusters. Each cluster tests a specific ontological relation using multiple semantically equivalent formulations. This structure enables systematic evaluation of whether a model produces consistent answers across paraphrases of the same underlying query.

Clusters cover several forms of graph-based reasoning, ranging from direct edge lookup to hierarchical, inverse, and relational reasoning. Each cluster is associated with an expected binary answer (yes or no) determined from the underlying ontology.

Each cluster identifier encodes the source node, cluster type, expected answer, and a unique identifier. The cluster type indicates the graph or logical pattern being tested, such as a direct edge, a transitive path, or an inverse relation.

Each cluster contains:

• the expected answer;
• the source node;
• the relation being tested;
• the target node, not necessarily adjacent in the case of inferred links.
• four paraphrased questions expressing the same underlying query; and
• four corresponding declarative statements.

The example below tests whether Application Configuration Database Record is a subConceptOf Configuration Database Record. Because this relation is supported by the ontology, the expected answer is yes.

{
  "Application_Configuration_Database_Record_edge_yes_2": {
    "expected_answer": "yes",
    "source": "Application Configuration Database Record",
    "property": "subConceptOf",
    "target": "Configuration Database Record",
    "questions": [
      "is 'Application Configuration Database Record' a subconcept of 'Configuration Database Record' ?",
      "is 'Application Configuration Database Record' a type of 'Configuration Database Record' ?",
      "is every kind of 'Application Configuration Database Record Configuration Database Record' also a kind of 'Configuration Database Record' ?",
      "is 'Application Configuration Database Record' a subcategory of 'Configuration Database Record' ?"
    ],
    "statements": [
      "'Application Configuration Database Record' is a subconcept of 'Configuration Database Record' ",
      "'Application Configuration Database Record' is a type of 'Configuration Database Record'",
      "every kind of 'Application Configuration Database Record Configuration Database Record' is also a kind of 'Configuration Database Record' ",
      "'Application Configuration Database Record' is a subcategory of 'Configuration Database Record' "
    ]
  }
}

The following example illustrates a negative cluster. Here, the ontology does not support the queried subConceptOf relation between Access Modeling and Identifier Activity Analysis, so the expected answer is no. For negative clusters, the corresponding declarative statements express the negated form of the queried relation.

"Access_Modeling_edge_no_1": {
      "expected_answer": "no",
      "source": "Access Modeling",
      "property": "subConceptOf",
      "target": "Identifier Activity Analysis",
      "questions": [
        "is 'Access Modeling' a subconcept of 'Identifier Activity Analysis' ?",
        "is 'Access Modeling' a type of 'Identifier Activity Analysis' ?",
        "is every kind of 'Access Modeling' also a kind of 'Identifier Activity Analysis' ?",
        "is 'Access Modeling' a subcategory of 'Identifier Activity Analysis' ?"
      ],
      "statements": [
        "'Access Modeling' is not a subconcept of 'Identifier Activity Analysis' ",
        "'Access Modeling' is not a type of 'Identifier Activity Analysis'",
        "not every kind of 'Access Modeling' is also a kind of 'Identifier Activity Analysis' ",
        "'Access Modeling' is not a subcategory of 'Identifier Activity Analysis' "
      ]
    }