Cloud Syslog Source

You can configure a cloud syslog source to allow a syslog client to send RFC 5424-compliant messages to Sumo. A source-specific token, generated by Sumo, is inserted into each message to identify the source. Transport-level security (TLS) 1.2 over TCP is required.

Syslog messages must be compliant with RFC 5424 or they are dropped. Messages over 64 KB are truncated.

Sumo manages an elastic scaling set of syslog servers, which scales up and down behind a set of AWS Elastic Load Balancers. The AWS ELB set can also scale up and down. For this reason, instead of IP address-based endpoints, Sumo uses endpoint hostnames in this format:

syslog.collection.YOUR_DEPLOYMENT.sumologic.com

where YOUR_DEPLOYMENT is au, ca, de, eu, fed, in, jp, kr, us1, or us2. For more information, see Sumo Logic Endpoints and Firewall Security.

info

FIPS 140-2 compliance is not available for Cloud Syslog in the FedRAMP deployment. It is with great emphasis that you must recognize and understand that the responsibility to mitigate information spillage is solely yours. We have no insight into your data or how it is classified.

In the procedure below, you configure a Cloud Syslog Source, this will generate a Sumo Logic token and the endpoint hostname. Then you set up TLS by downloading a cert to your server. Download the DigiCert certificate from one of the following locations:

• https://www.digicert.com/CACerts/DigiCertHighAssuranceEVRootCA.crt
• https://www.digicert.com/CACerts/DigiCertHighAssuranceEVRootCA.crt.pem

Sumo Logic supports syslog clients, including syslog-ng and rsyslog. Follow the instructions in the appropriate section below to configure your server to send syslog data. If syslog data does not appear in Sumo Logic, refer to Troubleshooting below.

Configure a Cloud Syslog Source

Cloud syslog configuration requires a token that is automatically generated when you configure a cloud syslog source. The token allows Sumo to distinguish your log messages from those of other customers. The token is tied to the source, but not to any specific user. 

Include the token as the Structured ID in every syslog message that is sent to Sumo Logic. The token is removed by Sumo Logic during ingestion and is not included with your syslog message in search results.

The token is deleted if you delete the source. To change a token, use the Regenerate Token option as described in the following procedure.

To configure a cloud syslog source, do the following:

1. In Sumo Logic select Manage Data > Collection > Collection.

2. On the Collection page, click Add Source next to a Hosted Collector. See Set up a Hosted Collector for information on adding Hosted Collectors.

3. Select Cloud Syslog.

4. Enter a Name to display for this source in Sumo. Description is optional.

5. (Optional) For Source Host and Source Category, enter any string to tag the output collected from this source. (Category metadata is stored in a searchable field called _sourceCategory.)

6. Fields. Click the +Add Field link to define the fields you want to associate, each field needs a name (key) and value.

   •  A green circle with a check mark is shown when the field exists in the Fields table schema.
   •  An orange triangle with an exclamation point is shown when the field doesn't exist in the Fields table schema. In this case, an option to automatically add the nonexistent fields to the Fields table schema is provided. If a field is sent to Sumo that does not exist in the Fields schema it is ignored, known as dropped.

7. Set any of the following under Advanced:

   • Enable Timestamp Parsing. This option is selected by default. If it's deselected, no timestamp information is parsed.
   • Time Zone. There are two options for Time Zone. You can use the time zone present in your log files, and then choose an option in case time zone information is missing from a log message. Or, you can have Sumo Logic completely disregard any time zone information present in logs by forcing a time zone. It's important to have the proper time zone set, no matter which option you choose. If the time zone of logs cannot be determined, Sumo Logic assigns the UTC time zone; if the rest of your logs are from another time zone your search results will be affected.
   • Timestamp Format. By default, Sumo will automatically detect the timestamp format of your logs. However, you can manually specify a timestamp format for a source. See Timestamps, Time Zones, and Time Ranges, and Date Formats.

8. Create any Processing Rules you'd like for the new source.

9. Click Save. The token information is displayed in a read-only dialog box, shown below.

    

10. Click Copy to copy the information for use in the syslog client. The information is copied in the following format:

    Token: 9HFxoa6+lXBmvSM9koPjGzvTaxXDQvJ4POE/WCURPAo+w4H7PmZm8H3mSEKxPl0Q@41123, Host: syslog.collection.YOUR_DEPLOYMENT.sumologic.com, TCP TLS Port: 6514

    The number 41123 in the token is the Sumo Private Enterprise Number (PEN). There are two options for including the token. You can include it in the structured data field or in the message body.  In the following example, the token is in the structured data field. 

    <165>1 2015-01-11T22:14:15.003Z mymachine.example.com evntslog - ID47 [YOUR_TOKEN] msg

    In the following example, the token is in the message body.

    <165>1 2015-01-11T22:14:15.003Z mymachine.example.com evntslog - ID47 - YOUR_TOKEN msg
    info

    RFC 5424 limits the structured data field (SD-ID) to 32 characters, however our token is 64 characters long. If your logging client enforces this limit you will need to pass the token in the message body.

11. After configuring the source, you can perform these token operations from the Collectors and Sources page:

    • Click Show Token to display the token for a cloud syslog source at any time. 
    • Click Regenerate Token if you need to generate a new token.

Syslog Clients

Follow the configuration instructions for your syslog client:

• syslog-ng
• rsyslog

Message format

Syslog messages must be in RFC 5424-compliant form or they are dropped. Messages over 64KB in length are truncated.

This diagram shows the RFC 5424 format:

Known issues

TCP connections go through Amazon Elastic Load Balancer (ELB) with an idle timeout set to 5 minutes. Connections are closed if they are idle for more than 5 minutes. It is possible to lose a few messages when the syslog client tries to re-establish the connection following an idle timeout.

Rsyslog provides the following configuration parameters to minimize data loss.

$ActionResumeRetryCount 1 # Reconnect but misses the first message after connection is idle for an hour.
$RebindInterval 1 #Establish a new connection every message. Change 1 to another number N so a new connection gets used every N messages.

If syslog messages fail to authenticate to the syslog cloud source—for example, if the token is missing or the source is deleted—the messages are dropped.

Troubleshooting

If you encounter problems, follow the instructions below to first verify the Sumo service connection, and then check the client configuration is correct.

Verify connection with Sumo service

To verify that the Sumo service can receive syslog messages, use a networking utility that supports TLS, such as nMap.org's ncat, to check that the syslog port accepts messages. 

$ ncat --ssl syslog.collection.YOUR_DEPLOYMENT.sumologic.com PORT

where YOUR_DEPLOYMENT is your deployment. For example:

$ ncat --ssl syslog.collection.us2.sumologic.com 6514

Then, enter a test message, for example:

<165>1 2017-10-24T06:00:15.003Z mymachine.example.com evntslog - ID47 - YOUR_TOKEN This is a message

where YOUR_TOKEN is the token that Sumo generated when you created the Cloud Syslog Source above.

Verify client configuration

If you followed the documentation for syslog-ng or rsyslog configurations but still do not see data in Sumo Logic, follow these steps to troubleshoot:

1. Verify that your cloud syslog token is valid, contains the ‘@41123’ part, and is in the correct location of your message (in the Structured Data field or the message body).

2. After each configuration change, restart your syslog client.

   $ sudo service rsyslog restart

   or

   $ /etc/init.d/syslog-ng restart

3. Verify that your syslog client is actually receiving data from your data sources.

4. Verify that there no firewall rules are blocking the packets.

5. Verify that your account is not throttled.

6. Run tcpdump to check if packets are flowing in both directions:

   $ sudo tcpdump host syslog.collection.YOUR_DEPLOYMENT.sumologic.com

       
   The output should look something like this:

   For TLS handshake:

   21:53:08.005676 IP client-host.client-port > server-host: Flags [S], seq 471724626, win 26883, options [mss 8961,sackOK,TS val 523109896 ecr 0,nop,wscale 7], length 0
   
   21:53:08.006321 IP server-host > client-host.client-port: Flags [S.], seq 3885139041, ack 471724627, win 28960, options [mss 1460,sackOK,TS val 112542097 ecr 523109896,nop,wscale 8], length 0
   
   21:53:08.006349 IP client-host.client-port > server-host: Flags [.], ack 1, win 211, options [nop,nop,TS val 523109896 ecr 112542097], length 0 ...

   For sending data:

   21:46:35.954790 IP client-host.client-port > server-host: Flags [P.], seq 471728183:471728572, ack 3885141774, win 256, options [nop,nop,TS val 523153927 ecr 112585055], length 389
   
   21:46:35.955326 IP server-host > client-host.client-port: Flags [.], ack 389, win 181, options [nop,nop,TS val 112586129 ecr 523153927], length 0 ...

7. If the previous steps do not solve the problem, file a support case with information about the specific Collector and Source setup. You can also include the tcpdump if available.