Skip to main content

2023 Cloud SIEM Signal Index Migration FAQ

Summary​

Effective in 2022, Signals generated by Cloud SIEM are automatically saved in a standardized sec_signal index. This special partition is similar to the existing sec_record indices in that, unlike data retained using the older Signal Forwarding feature, it is stored in a format that supports keyword search, nested attributes, and other standard log search features.

The new index is automatically generated and retained for a period of 2 years at no additional cost for all Cloud SIEM customers.

As a result, the optional Signal Forwarding feature in Cloud SIEM is deprecated on November 15, 2023. Existing data is not deleted, but new Signals generated after that date are no longer forwarded and the option is no longer available in Cloud SIEM. Customers leveraging Signal Forwarding data to generate dashboards (or for other use cases) will need to modify those applications to use the new sec_signal index before then.

Differences compared to Signal Forwarding​

The new sec_signal index has some differences compared to the older Signal Forwarding feature. The new index:

  • Fully supports log search features such as keyword search, nested attributes, filtering by selected values, and so on.
  • Has standardized attribute names/formats (in alignment with standard Sumo Logic naming conventions as used in the sec_record indices). This goes beyond just the switch from “snake_case” to “camelCase”; specifically:
    • The asset object has been replaced with the more accurate entities object. The new object is (potentially) multi-valued, and the attribute names have changed from name to value, from asset_type to type. Also, entity_criticality (now criticality) and entity_suppressed (now suppressed) have moved from the top level of the log message to the entities object. These values can be accessed in an expression with syntax like | count by %entities[0].type.
    note

    As a reminder (this is not new with the new index), criticality and suppression are representative of those values at the point in time that the Signal was created. Those values could have changed later.

    • Timestamps are now strings instead of integers.
    • The prototype attribute is now a string instead of a boolean.
    • The extra object has been deprecated. The matchExpression attribute contained in this object has been promoted to the top level of the log message.
  • The following attributes/objects are no longer included in the new index:
    • is_whitelisted (which was deprecated in April 2022)
    • multiEntitySharedId (this was an internal attribute that was not meant to be exposed to users)
    • new_rule_id (duplicate of rule_id)
    • The threat_indicators object is no longer included (this has been deprecated for some time)
  • Users may see a different number of signals for the same time period when comparing the sec_signal index with the old Signal Forwarding data. This is due to a number of factors related to how the Sumo platform and Cloud SIEM calculate start and end times and which timestamps are used in the different indices. But over the long term, both methods include the same signals.

The following code examples show the same Signal in the old format from Signal Forwarding and the new format in the sec_signal index:

Old format​

{
	asset: {
		name:"vagrant",
		is_whitelisted:false,
		asset_type:"username",
		id:"_username-vagrant"
	},
	attack_stage:"Initial Access",
	content_type:"RULE",
	created_timestamp:1676492228663,
	description:"Failed Authentication Rule",
	entity_criticality:"Critical Asset",
	entity_id:"_username-vagrant",
	entity_suppressed:true,
	extra: {
		matchExpression:"metadata_deviceEventId='Security-4625'"
	},
	full_records: [
		{
			action:"Status OK",
			application:"",
			description:"An account failed to log on",
			device_hostname:"dc",
			device_hostname_raw:"dc.windomain.local",
			dstDevice_hostname:"dc",
			dstDevice_hostname_raw:"dc.windomain.local",
			errorCode:"0x0",
			errorText:"Status OK",
			fieldTags:{ },
			fields: {
				EventData.WorkstationName:"DC",
				EventData.ProcessId:"0x0",
				EventData.KeyLength:"0",
				Task:"12544",
				Keywords:"Audit Failure",
				EventData.TargetDomainName:"windomain.local",
				EventData.Status:"0xc000006d",
				Provider.Guid:"{54849625-5478-4994-A5BA-3E3B0328C30D}",
				EventData.AuthenticationPackageName:"NTLM",
				Opcode:"Info",
				Provider.Name:"Microsoft-Windows-Security-Auditing",
				EventData.IpAddress:"127.0.0.1",
				TimeCreated:"2023-02-15T20:06:49.5088771Z",
				EventData.TargetUserSid:"S-1-0-0",
				Version:"0",
				EventData.LogonType:"3",
				EventData.SubStatus:"0x0",
				Computer:"dc.windomain.local",
				Correlation.ActivityID:"{1083FC66-AA4D-0000-85FC-83104DAAD701}",
				Execution.ProcessID:"612",
				TimeCreated.SystemTime:"2023-02-15T20:06:49.5088771Z",
				Execution.ThreadID:"3532",
				EventData.FailureReason:"%%2304",
				Channel:"Security",
				EventRecordID:"65378267",
				EventData.SubjectUserSid:"S-1-0-0",
				EventData.TargetUserName:"vagrant",
				EventID:"4625",
				Level:"Information",
				EventData.IpPort:"51934",
				EventData.SubjectLogonId:"0x0"
			},
			http_requestHeaders:{ },
			listMatches:[ ],
			logonType:"Network",
			matchedItems:[ ],
			metadata_deviceEventId:"Security-4625",
			metadata_mapperName:"Windows - Security - 4625",
			metadata_mapperUid:"0e3b7ced-f6b4-49a0-8fbc-8128ccdb4670",
			metadata_orgId:"000000000025BD5C",
			metadata_parseTime:1676492227559,
			metadata_product:"Windows",
			metadata_productGuid:"1ff7546c-cb36-4a24-87f7-89d2cecc5761",
			metadata_sensorZone:"default",
			metadata_sourceCategory:"cse/windows/event",
			metadata_sourceMessageId:"2373747457122604377",
			metadata_vendor:"Microsoft",
			normalizedAction:"domainLogon",
			objectType:"Authentication",
			srcDevice_hostname:"dc",
			srcDevice_hostname_raw:"DC",
			srcDevice_ip:"127.0.0.1",
			srcDevice_ip_ipv4IntValue:2130706433,
			srcDevice_ip_isInternal:false,
			srcDevice_ip_version:4,
			success:false,
			timestamp:1676491609508,
			uid:"044922c0-6044-5fa8-b66c-914a04140572",
			user_authDomain:"windomain.local",
			user_username:"vagrant",
			user_username_raw:"vagrant",
			objectClassification:"Authentication"
		}
	],
	id:"ecf375e0-1ab2-58f3-8742-96004203b653",
	multiEntitySharedId:"41c9c3d4-da3d-4e13-a95f-e1148152c176",
	name:"Failed Authentication Rule",
	new_rule_id:"MATCH-U00061",
	objectClassification:"Signal"
	prototype:false,
	rule_id:"MATCH-U00061",
	rule_name:"Failed Authentication Rule",
	severity:5,
	summary:"Failed Authentication Rule",
	suppressed_reasons: [
		"entity:_username-vagrant"
	],
	tags: [
		"_mitreAttackTactic:TA0001",
		"_mitreAttackTechnique:T1087"
	],
	timestamp:"1676491609508",
	threat_indicators: [
		{
			indicator_type:"ip",
			value:"127.0.0.1"
		}
	],
}

New format​

{
	attackStage:"Initial Access",
	contentType:"RULE",
	createdTimestamp:"1676492228663",
	description:"Failed Authentication Rule",
	entities: [
		{
			criticality:"Critical Asset",
			id:"_username-vagrant",
			suppressed:true,
			type:"_username",
			value:"vagrant",
		}
	],
	fullRecords: [
		{
			action:"Status OK",
			application:"",
			description:"An account failed to log on",
			device_hostname:"dc",
			device_hostname_raw:"dc.windomain.local",
			dstDevice_hostname:"dc",
			dstDevice_hostname_raw:"dc.windomain.local",
			errorCode:"0x0",
			errorText:"Status OK",
			fields: {
				EventData.WorkstationName:"DC",
				EventData.ProcessId:"0x0",
				EventData.KeyLength:"0",
				Task:"12544",
				Keywords:"Audit Failure",
				EventData.TargetDomainName:"windomain.local",
				EventData.Status:"0xc000006d",
				Provider.Guid:"{54849625-5478-4994-A5BA-3E3B0328C30D}",
				EventData.AuthenticationPackageName:"NTLM",
				Opcode:"Info",
				Provider.Name:"Microsoft-Windows-Security-Auditing",
				EventData.IpAddress:"127.0.0.1",
				TimeCreated:"2023-02-15T20:06:49.5088771Z",
				EventData.TargetUserSid:"S-1-0-0",
				Version:"0",
				EventData.LogonType:"3",
				EventData.SubStatus:"0x0",
				Computer:"dc.windomain.local",
				Correlation.ActivityID:"{1083FC66-AA4D-0000-85FC-83104DAAD701}",
				Execution.ProcessID:"612",
				TimeCreated.SystemTime:"2023-02-15T20:06:49.5088771Z",
				Execution.ThreadID:"3532",
				EventData.FailureReason:"%%2304",
				Channel:"Security",
				EventRecordID:"65378267",
				EventData.SubjectUserSid:"S-1-0-0",
				EventData.TargetUserName:"vagrant",
				EventID:"4625",
				Level:"Information",
				EventData.IpPort:"51934",
				EventData.SubjectLogonId:"0x0"
			},			
			fieldTags:{ },
			http_requestHeaders:{ },
			listMatches:[ ],
			logonType:"Network",
			matchedItems:[ ],
			metadata_deviceEventId:"Security-4625",
			metadata_mapperName:"Windows - Security - 4625",
			metadata_mapperUid:"0e3b7ced-f6b4-49a0-8fbc-8128ccdb4670",
			metadata_orgId:"000000000025BD5C",
			metadata_parseTime:1676492227559,
			metadata_product:"Windows",
			metadata_productGuid:"1ff7546c-cb36-4a24-87f7-89d2cecc5761",
			metadata_sensorZone:"default",
			metadata_sourceCategory:"cse/windows/event",
			metadata_sourceMessageId:"2373747457122604377",
			metadata_vendor:"Microsoft",
			normalizedAction:"domainLogon",
			objectType:"Authentication",
			srcDevice_hostname:"dc",
			srcDevice_hostname_raw:"DC",
			srcDevice_ip:"127.0.0.1",
			srcDevice_ip_ipv4IntValue:2130706433,
			srcDevice_ip_isInternal:false,
			srcDevice_ip_version:4,
			success:false,
			timestamp:1676491609508,
			uid:"044922c0-6044-5fa8-b66c-914a04140572",
			user_authDomain:"windomain.local",
			user_username:"vagrant",
			user_username_raw:"vagrant",
			objectClassification:"Authentication"
		}
	],
	id:"ecf375e0-1ab2-58f3-8742-96004203b653",
	matchExpression:"metadata_deviceEventId='Security-4625'",
	name:"Failed Authentication Rule",
	objectClassification:"Signal",
	prototype:"false",
	ruleId:"MATCH-U00061",
	ruleName:"Failed Authentication Rule",
	severity:"5",
	summary:"Failed Authentication Rule",
	suppressedReasons: [
		"entity:_username-vagrant"
	],
	tags: [
		"_mitreAttackTactic:TA0001",
		"_mitreAttackTechnique:T1087"
	],
	timestamp:"1676491609508"
}

Migrating from Signal Forwarding to the new index​

Because the signal data is used differently in different applications, it’s not possible to outline an exact set of steps to take to migrate to the new index. However, in general, the primary step is to adapt to the new attribute names and related changes listed above. For example, if retrieving the entity criticality, one must switch from the entity_criticality attribute to the %entities[0].criticality attribute.

Multi-record Signals (such as those generated by Threshold rules) only include 1 attached record. You will need to perform a log search (in the sec_record* indices) to find any additional related records, based on the rule expression for that Signal.

Note on Fields section in attached records​

Prior to November 6, 2023, the new sec_signal objects did not include the fields{} section for each record. However, that has now changed; the fields{} section is now included.

Status
Legal
Privacy Statement
Terms of Use

Copyright © 2024 by Sumo Logic, Inc.