|
|||||
 |
 |
||||
| CEE Website is in "Archive" status — read the announcement | |||||
|---|---|---|---|---|---|
| CEE Language |
|---|
Additional Information |
CEE Log Transport (CLT) Specification1.0-beta1
- Introduction
- Protocol Requirements
- CTL Implementation Requirements
- Transport Mappings
- Appendix 1: CEE Over Syslog Transport Mapping
This document describes the CEE Common Log Transport (CLT) requirements that define the mandatory and preferred capabilities for a log transport protocol. Such protocols enable CEE Log Syntax (CLS) encoded event records to be shared between parties in a universal, machine-readable manner. The intent of CLT is to provide guidance for vendors and end users regarding how event records should be reliably and securely shared.
1 Introduction
The CEE Log Transport (CLT) provides the technical support necessary for a secure, interoperable, and reliable log infrastructure. A log infrastructure requires more than just standardized event records, support is needed for international string encodings, secure logging services, standardized event interfaces, and secure, verifiable log trails.
The CLT defines a listing of requirements that a CLT Protocol must meet to be considered CEE-compatible as well as additional capabilities that may be required for specific use cases. For example, a CLT-compatible Protocol must be able to transmit a Common Log Syntax (CLS) encoded CEE Event. More advanced CLT Protocols may provide additional capabilities such as encryption and full acknowledgments. A CLT Protocol may also be able to specify or transmit CEE Profiles and additional event-related information, such as packet captures or file data.
CLT also defines transport mappings. A CLT Mapping defines a standardized way for CEE Events to be transmitted over a certain CLT Protocol. CLT Mappings allow for consistency in using the specified transport mechanism to transmit CEE events: for example, it may specify flags to indicate the data is a CEE event or the character encoding used. To date, the only mapping published by CEE is a specification for sending JSON-encoded events over the RFC5425 TLS Syslog protocol.
2 Protocol Requirements
Many uses for events records require them to be transported from the originating system. CLT Protocols must meet some basic requirements in order to do this reliably and efficiently. This document categorizes these requirements into four groups based on the requirements of the use case. Conformance Level 0 is the mandatory conformance level and includes basic capability. Further conformance levels describe optional capabilities that more advanced CLT protocols should support. Conformance Level 1 is core capability providing the minimum set of requirements for robustness. Conformance Level 2 contains a set of additional requirements that address logging in the presence of attackers. Conformance Level 3 contains an additional set of requirements that address local administrative attack scenarios. Conformance Level 3 is the most robust requirements set.
2.1 Conformance Level 0 - Core Requirements
2.1.1 0.1 Publish
The CLT Protocol shall be a published protocol specification with no licensing barriers to interoperability, no royalties and no approval process. The CLT Protocol shall utilize only those protocols and standards which are openly available.
2.1.2 0.2 Transport
The CLT Protocol shall be able to transport at least one form of CEE encoded event records within the body of the protocol packet.
2.1.3 0.3 Self-Identification
The CLT Protocol shall make evident by the protocol as to which data belongs to CEE event records and the CEE defined encoding used when a transmission is made consisting of CEE and non-CEE data.
2.1.4 0.3.1 Identification of CEE Events
The CLT Protocol shall have a mechanism that identifies the CEE Events within the protocol packet body.
2.1.5 0.3.2 Encoding Identifier
A CLT Protocol capable of handling more than one (1) data encoding method shall provide evidence as to the encoding being used.
2.1.6 0.4 Timestamp
Each protocol packet shall have timestamp that indicates the date and time the packet was transmitted. A valid timestamp must indicate year, month, day, hour, minute, and second. Including sub-seconds is encouraged.
2.2 Conformance Level 1 - Basic Capabilities
One of the core capabilities is for the log transport to perform in limited bandwidth environments. Additional core capabilities include log transport tamper detection.
2.2.1 1.1 Event Record Delivery
The CLT Protocol shall preserve the integrity of the logical order of a channel's packets such that the Event Receiver will be able to reconstitute the original logical order.
2.2.2 1.2 Compression of Records
The CLT Protocol shall provide a method that allows the Event Sender to compress the packet body prior to transmission.
2.2.3 1.3 Missing Record Detection
The CLT Protocol shall be able to accurately and reliably detect missing transport packets.
2.2.4 1.4 Message Identification
The CLT Protocol shall support message identifiers.
2.2.5 1.4.1 Packet Duplication
The CLT Protocol shall be able to identify when duplicate packets have been received.
2.2.6 1.4.2 Packet Acknowledgement
The CLT Protocol shall support at least one method of allowing the Event Receiver to acknowledge (ACK) that a packet has been received.
2.2.7 1.4.3 Packet Retransmission
The CLT Protocol shall be able to retransmit individual packets on request at least until the Event Receiver has acknowledged reception or until a nominal time has elapsed.
2.2.8 1.5 Packet Traversal Traceability
The CLT Protocol shall be capable of tracing and recording the path a packet traverses. The intent of this requirement is to allow packets to be traced through Session Relay devices such as NATs.
2.2.9 1.6 Tamper Detection
The CLT Protocol shall accurately and reliably detect any evidence of tampering or data corruption through the use of digital signatures or other anti-tamper mechanisms.
2.3 Conformance Level 2 - Securely Log in the Presence of Attackers
The core theme for the Conformance Level 2 transport requirements is the capability for CLT Protocol to securely log in the presence of attackers. Additional requirements include Confidentiality, Authenticity, and transmission Encryption.
2.3.1 2.1 Full Integrity Acknowledgements
The CLT Protocol shall be capable of producing packet level acknowledgements that contains data through which the sender can verify the integrity of a received packet. This requirement is an extension of that of Requirement 1.4.2: Packet Acknowledgement.
2.3.2 2.2 Message Replay Protection
The CLT Protocol shall protect against message replay.
2.3.3 2.3 Event Integrity
The CLT Protocol shall accurately and reliably detect any repeated or unexpected message identifiers.
2.3.4 2.3.1 Chain of Modification
The CLT Protocol shall maintain the chain of modification of CEE Event data that is modified while in transit.
2.3.5 2.3.2 Reproduction of Original Event
The CLT Protocol shall be able to reproduce by request, the original CEE Event that is modified while in transit.
2.3.6 2.4 Transmission Encryption
The CLT Protocol shall support transmission encryption using best practice encryption algorithms. One way to achieve this requirement is by using Transport Layer Security (TLS).
2.3.7 2.5 Confidentiality
The CLT Protocol shall maintain confidentiality of data, minimally within the packet body. The CLT Protocol cryptography modules shall be capable of performing data encryption using data encryption best practices.
2.3.8 2.6 Authenticity
The CLT Protocol shall maintain authenticity of the data in transit.
2.3.9 2.6.1 Use of SASL, GSS-API, and Kerberos
The CLT Protocol shall support authenticity using Simple Authentication and Security Layer (SASL), Generic Security Services Application Program Interface (GSS-API), and Kerberos.
2.4 Conformance Level 3 - Secure Against Local Administration Attacks
The core theme for the Conformance Level 3 transport requirements is the capability for CLT Protocol to securely log in the presence of local administration attacks.
2.4.1 3.1 Tamper Resistant
The CLT Protocol shall maintain integrity mechanisms resistant to tampering by local administrator (e.g. perfect forward secrecy).
2.4.2 3.2 Record Channels
The CLT Protocol shall be able to provide support for multiple channels within a session.
2.4.3 3.3 Profile Channels
A CLT Protocol channel shall be able to have CEE-specific metadata bound to it, such as a CEE Event Profile. This metadata can be used by the Event Sender/Receiver to exchange record format data or reduce duplicative data from being sent.
3 CTL Implementation Requirements
3.1 Conformance Level 0 - Core Requirements
3.1.1 0.1 Support CLT Protocol Level 0
The implementation must support at least a Conformance Level 0 of the CLT Protocol.
3.2 Conformance Level 1 - Basic Requirements
3.2.1 1.1 Support CLT Protocol Level 1
The implementation must support at least a Conformance Level 1 of the CLT Protocol.
3.2.2 1.2 Sender-side Buffering
The CLT Protocol shall be capable of sender-side buffering, e.g. event record is retained in a recoverable fashion on sender until server indicates that event has been received.
3.2.3 1.2.1 Single Log Record Buffering
The client shall retain each log record until the server has acknowledged (ACKed) the reception of the message; ideally this ACK should include a hash or signature whereby the sender can validate the message was correctly received.
3.2.4 1.2.2 Batch Log Record Buffering
The client shall retain each batch of log records until the server has ACKed the reception of the batch message; ideally this ACK should include a hash or signature whereby the sender can validate the message was correctly received.
3.2.5 1.2.3 Enable and Disable Switch
The CLT Protocol shall have a switch that enables and disables send-side buffering.
3.2.6 1.3 Log in Limited Network Environments
The CLT Protocol shall be capable of reordering the event transmission queue so that the most important messages receive priority. Many environments utilize Network Address Translation (NAT). The CLT Protocol shall be capable of functioning correctly in that environment.
3.2.7 1.3.1 Retransmission Priority
The Priority of the event retransmission queue shall be settable by the application.
3.2.8 1.3.2 Network Address Translation (NAT)
The CLT Protocol shall be capable of communicating in a Network Address Translation (NAT) environment.
3.3 Conformance Level 2 - Log In The Presence of Attackers
3.3.1 2.1 Support CLT Protocol Level 2
The implementation must support at least a Conformance Level 2 of the CLT Protocol.
3.4 Conformance Level 3 - Secure Against Local Administration Attacks
3.4.1 3.1 Support CLT Protocol Level 3
The implementation must support at least a Conformance Level 3 of the CLT Protocol.
3.4.2 3.2 Secure against local administration attacks
The CLT Protocol shall, on average, transmit event records within 1 second of event record creation.
3.4.3 3.3 Event Source Channel Binding
Applications sending events over the CLT Protocol shall be able to bind the event records to a CLT Protocol channel.
3.4.4 3.4 Event Destination Channel Binding
Applications receiving events over a CLT Protocol channel shall be able to reconstruct the full event record based on the channel contents and previously exchanged channel metadata.
3.4.5 3.5 Channel Profiles
Applications shall be able to bind one or more event profiles to a registered channel.
3.4.6 3.6 Continuous Operation
The CLT Protocol shall be able to support load balancing and gracefully failover to backup servers when the primary is lost.
4 Transport Mappings
Transport mappings specify standard interoperability mapping between CEE and popular log transmission protocols. Mappings have been defined between CEE and the following protocol:
- Syslog: Conformance Level 0
See the associated appendix for information on each mapping.
5 Appendix 1: CEE Over Syslog Transport Mapping
5.1 Introduction
Since Syslog is the de facto standard in log transport protocols and it is supported by numerous products, CEE provides a way to send CEE Event data over Syslog. This document defines a standard process to encode CEE Events using a CEE Log Syntax (CLS) Encoding and place the encoded event into a Syslog message.
5.2 Syslog
There exist several different formats for Syslog messages. Within this document, they will be separated into two groups: legacy and RFC5424. While the basic process for transmitting CEE Events is the same for both Syslog groups, there are some nuances surrounding the use of legacy Syslog of which implementers must be cognizant.
5.2.1 Legacy Syslog
Most people are familiar with legacy Syslog. Legacy Syslog is characterized by its timestamp format missing the year and the event message consisting only of an unstructured line of text. The various formats of legacy Syslog messages are identified in RFC3164.
Besides the variance in message formats, not all legacy Syslog implementations can handle 8-bit character sets. That is, some implementations use only the lower 7 bits of each byte. This causes problems when trying to send binary data or extended character sets (e.g., extended ASCII, UTF-8) that rely on "8-bit clean" processing.
5.2.2 RFC5424 Syslog
The Syslog specification was updated in March 2009 with the release of RFC5424. This update brought many needed enhancements, including a standardized timestamp that includes the year, Unicode support (UTF-8), and a way of providing more structured content within Syslog messages.
One of the lesser recognized improvements was to make the entire Syslog protocol and RFC5424 implementations 8-bit clean. This step was necessary to enable full Unicode support.
While RFC5424 Syslog is preferred over legacy versions, there are still many environments and platforms that have been built on top of legacy Syslog implementations. All Syslog implementations supporting CEE SHOULD be RFC5424-compatible.
5.3 CLT Syslog Mapping
This document defines a CLT mapping that is conformant to both CEE and Syslog specifications via the inclusion of a CLS JSON (Javascript Object Notation) encoded event within a Syslog message.
Syslog Header
The standard Syslog header MUST be used. The actual formatting of the Syslog header is dependent on the Syslog protocol version and may vary based on the implementation. Regardless, these header values are for the Syslog protocol and are independent of CEE. The Syslog header values SHOULD NOT be used to add or modify any values within an enclosed CEE Event.
Syslog Body
The CEE Event MUST be represented using a CLS Encoding. For compliance with this specification, the CEE Event MUST be represented using the CLS JSON Encoding.
In contrast to Section 2 of RFC4627, the JSON encoding suitable for Syslog transport MUST NOT contain insignificant whitespace before or after any of the six JSON structural characters.
CEE Event Flag
The beginning of the encoded CEE Event MUST be identified by the CEE Event Flag. Within Syslog, the CEE Event Flag is @cee: (ABNF %x40.63.63.65.3A).
The CEE Event Flag MAY be followed by one space (' ', ABNF %x20) character.
A CLS JSON Encoded CEE Event MUST appear immediately following a CEE Event Flag. Other, non-CEE, non-JSON content MUST NOT appear in the Syslog body after a CEE Event Flag. A Syslog message MUST NOT contain more than one CEE Event Flag.
Character Encoding
All CLS Encodings, including the CLS JSON Encoding, assume an 8-bit clean environment. Therefore, it is important that the event producer understand the Syslog format that will be used, especially whether implementation is a 7-bit or 8-bit clean implementation.
If the Syslog implementation is 8-bit clean, then the implementation supports UTF-8 and no additional encodings are necessary beyond those required by the JSON CLS Specification. However, if the implementation is only 7-bit, then all characters not in the ASCII character set (ABNF %x00-7F) MUST be additionally escaped.
When using a 7-bit Syslog implementation or have requirements for 7-bit Syslog compatibility, then all UTF-8 encoded characters that cannot be represented using the lower 7 bits of an 8-bit byte MUST be escaped. These characters SHOULD be escaped using the JSON escape sequences RFC4627, especially the Unicode escape: \u followed by four (4) hexadecimal digits (ABNF %x5C.75
4HEXDIG).
5.3.1 Transmission
A Syslog message containing a CEE Event should be able to be transmitted using any Syslog-based transport mechanism. Due to the potential priority or sensitivity of certain event records, it is recommended that the transmission protocol supply the necessary confidentiality and integrity measures for the event content and operating environment.
The original Syslog protocol sent event records in plaintext over UDP. This does not provide any security controls. One option is to transmit the Syslog messages using Transport Layer Security, as specified in RFC5425.
5.3.2 Examples
Example 1 - Valid
A valid CEE JSON Event Record embedded within an RFC5424 Syslog transport:
<165>1 2011-12-20T12:38:06Z 10.10.0.1 process - example-event-1 @cee:{"pname":"auth","host":"system.example.com","time":"2011-12-20T12:38:05.123456-05:00"}
Example 2 - Valid
A valid CEE JSON Event Record used with a "legacy" Syslog transport:
<0>Dec 20 12:42:20 syslog-relay process[35]: @cee: {"crit":123,"id":"abc","appname":"application","pname":"auth","pid":123,"host":"system.example.com","pri":10,"time":"2011-12-20T12:38:05.123456-05:00","action":"login","domain":"app","object":"account","service":"web","status":"success"}
Page Last Updated: August 09, 2012
