# Incident Response & Computer Forensics, Third Edition ![rw-book-cover](https://learning.oreilly.com/library/view/incident-response/9780071798686/ibis_generated_cover_thumbnail.jpg) ## Metadata - Author: - Full Title: Incident Response & Computer Forensics, Third Edition - Category: #books ## Highlights ### Introduction - Today, effective incident response requires more than just IT and security staff—legal, human resources, public relations, marketing, and other business functions are needed. - Write better reports - We discuss checklists, case notes, development of leads, creating indicators of compromise, and determining the scope of the incident. - Each incident you work on will require the collection and preservation of information. In this part, we discuss collecting data from both running and offline systems, the network, and from enterprise services. Data sources include memory, hard drives, network packet captures, and log files. ### Chapter 1: Real-World Incidents - However, the industry has changed, and the definition of an “incident” needs to change as well. We’d like to expand this definition to “any unlawful, unauthorized, or unacceptable action that involves a computer system, cell phone, tablet, and any other electronic device with an operating system or that operates on a computer network.” - As part of that definition, we also recommend that you review local laws that define computer crimes. - Prevent a disjointed, noncohesive response - In general, an incident response consists of an investigation team that determines what happened and performs a damage assessment, a remediation team that removes the attacker from the environment and enhances the victim’s security posture, and some form of public relations (to senior level management, internal employees, business partners, or the public). - Systems have more storage capacity than ever before, so capturing bit-by-bit forensic images of suspected compromised systems throughout a large incident is time consuming and difficult at best. - The intent of this book is to arm incident responders with the tools and techniques painstakingly developed and refined over the last ten years by the authors, who have responded to hundreds of large-scale, international, complex, and public incidents. - The attacker then executed the mimikatz.exe tool to extract Bob’s password and VPN machine certificate. - Before we end this chapter, we’d like to introduce a concept that we use to help explain the various phases of an attack. We call this concept the “attack lifecycle,” and we use it to demonstrate the various stages of most compromises. We use this concept heavily when planning remediation; however, we felt it belonged in this chapter so that we could use examples from the case studies you just read to illustrate each phase of the attack lifecycle. - The attackers accomplish their goal, which often includes the theft of data or modification of existing data. ### Chapter 2: IR Management Handbook - We mentioned earlier in this chapter that many of the challenges to effective incident response are nontechnical. Staying organized is one of those challenges, and is an especially big one. We hate to use the term “situational awareness,” but that’s what we are talking about here. Your investigations must have a mechanism to easily track critical information and share it with the ancillary teams and the organization’s leadership. - But we’ve learned some interesting things over our collective 30 years of responding to hundreds of incidents in organizations both large and small. With respect to this chapter, two specific points come to mind. First, most of the challenges to incident response are nontechnical. Second, the core principles of investigating a computer security incident do not differ from a nontechnical investigation. The real challenge is in looking past the buzzwords and marketing hype that attempts to convince you otherwise. - Defining what a computer security incident is establishes the scope of what your team does and provides focus. It is important to establish that definition, so everyone understands the team’s responsibilities. If you don’t already have one, you should create a definition of what “computer security incident” means to your organization. - If there is no intent to cause harm, it is hard to call an event an incident - Most organizations take a tiered and blended approach to staffing both investigative and remediation teams. Assembled and dedicated during the course of the investigation, the core teams often consist of senior IT staff, especially those with experience with log review, forensic analysis, and malware triage skills. The investigative team should have the ability to quickly access log repositories, system configurations, and, if an enterprise-wide IR platform is available, authority to conduct searches for relevant material. - Running investigations is a highly experiential skill that improves over time. - With the exception of the situation where a company has virtually no internal IT capabilities, we have found that organizations that assemble an IR team on their own, however informally, stand a better chance of a successful investigation and timely resolution. This is true, even if the IR team is set up to handle only the initial phases of an investigation while external assistance is engaged. - Many IR and forensic professionals of all skill levels lurk on message boards, such as Forensic Focus. - The fundamental skills that provide the greatest benefit to this field are the same as in most sciences: observation, communication, classification, measurement, inference, and prediction. The individuals possessing these skills typically make the best IR team members. - The incident response process consists of all the activities necessary to accomplish the goals of incident response. The overall process and the activities should be well documented and understood by your response team, as well as by stakeholders throughout your organization. The process consists of three main activities, and we have found that it is ideal to have dedicated staff for each: • Initial response • Investigation • Remediation - Initial response is an activity that typically begins the entire IR process. Once the team confirms that an incident is under way and performs the initial collection and response steps, the investigation and remediation efforts are usually executed concurrently. The investigative team’s purpose is solely to perform investigatory tasks. During the investigation, this team continually generates lists of what we call “leads.” Leads are actionable items about stolen data, network indicators, identities of potential subjects, or issues that led to the compromise or security incident. These items are immediately useful to the remediation team, whose own processes take a significant amount of time to coordinate and plan. In many cases, the activity that your team witnesses may compel you to take immediate action to halt further progress of an intrusion. - The goal of an investigation is to determine facts that describe what happened, how it happened, and in some cases, who was responsible. - Without knowing facts such as how the attacker gained access to your network in the first place, or what the attacker did, you are not in a good position to remediate. It may feel comforting to simply pull the plug and rebuild a system that contains malware, but can you sleep at night without knowing how the attacker gained access and what they did? Because we value our sleep, we’ve developed and refined a five-step process, shown in Figure 2-2, that promotes an effective investigation. - IOC (pronounced eye-oh-cee) creation is the process of documenting characteristics and artifacts of an incident in a structured manner. This includes everything from both a host and network perspective—things beyond just malware. Think about items such as working directory names, output file names, login events, persistence mechanisms, IP addresses, domain names, and even malware network protocol signatures. The goal of IOCs is to help you effectively describe, communicate, and find artifacts related to an incident. Because an IOC is only a definition, it does not provide the actual mechanism to find matches. You must create or purchase technology that can leverage the IOC language. - Using IOCs to document indicators is great, but their real power is in enabling IR teams to find evil in an automated fashion, either through an enterprise IR platform or through visual basic (VB) and Windows Management Instrumentation (WMI) scripting. The success of an investigation depends on your ability to search for IOCs across the enterprise and report on them in an automated way—this is what we mean by “IOC deployment.” Therefore, your organization must possess some capability to implement IOCs, or they are not much use. For network-based IOCs, the approach is straightforward—most solutions support Snort rules. However, as discussed earlier, there is not yet an accepted standard for host-based IOCs. Because of this, effectively using host-based IOCs in your investigative process may be challenging. Let’s take a look at some current options. - Memory collection is most useful in cases where you suspect the attacker is using a mechanism to hide their activities, such as a rootkit, and you cannot obtain a disk image. Memory is also useful for cases where the malicious activity is only memory-resident, or leaves very few artifacts on disk. In the majority of systems we respond to, memory is not collected, however. - During most investigations, we come across files that are suspected malware. We have a dedicated team of malware analysts that examines these files. - A forensic examination performed on disk images during an incident response is a very focused and time-sensitive task. When bullets are flying, you do not have time to methodically perform a comprehensive examination. We normally write down a handful of realistic questions we want answered, decide on an approach that is likely to uncover information to answer them, and then execute. - When performing intrusion analysis, remember that you may not “find all the evidence.” We’ve worked with organizations that suffer from what we sometimes call the “CSI effect,” where the staff believes they can find and explain everything using “cool and expensive tools.” We collectively have decades of experience performing hundreds of incident response investigations, and during all of these investigations, we have yet to come across such a magic tool. Granted, there are tools that can greatly benefit your team. However, some of the best tools to use are ones you already have—you are using them now, to read and understand this sentence. - We have found that the best time to begin remediation is when the detection methods that are in place enter a steady state. That is, the instrumentation you have configured with the IOCs stop alerting you to new unique events. - What is “significant investigative information”? We have found a handful of data points that are critical to any investigation. These items must be tracked as close to real time as possible, because team members will use them as the “ground truth” when it comes to the current status of the investigation. This data will also be the first thing that team members will reference when queries come in from management. • List of evidence collected  This should include the date and time of the collection and the source of the data, whether it be an actual person or a server. Ensure that a chain of custody is maintained for each item. Keep the chain of custody with the item, and its presence in this list is an indicator to you that an item has been handled properly. • List of affected systems  Track how and when the system was identified. Note that “affected” includes systems that are suspected of a security compromise as well as those simply accessed by a suspicious account. • List of any files of interest  This list usually contains only malicious software, but may also contain data files or captured command output. Track the system the file was found on as well as the file system metadata. • List of accessed and stolen data  Include file names, content, and the date of suspected exposure. • List of significant attacker activity  During examinations of live response or forensic data, you may discover significant activities, such as logins and malware execution. Include the system affected and the date and time of the event. • List of network-based IOCs  Track relevant IP addresses and domain names. • List of host-based IOCs  Track any characteristic necessary to form a well-defined indicator. • List of compromised accounts  Ensure you track the scope of the account’s access, local or domain-wide. • List of ongoing and requested tasks for your teams  During our investigations, we usually have scores of tasks pending at any point. From requests for additional information from the ancillary teams, to forensic examinations, it can be easy to let something fall through the cracks if you are not organized. - As consultants, our reports are fundamental deliverables for our customers. Creating good reports takes time—time you may think is best spent on other tasks. However, without reports, it is easy to lose track of what you’ve done. We’ve learned that even within a single investigation, there can be so many findings that communicating the totality of the investigation would have been difficult without formal, periodic reports. In many investigations, the high-level findings are based on numerous technical facts that, without proper documentation, may be difficult to communicate. We believe reports are also a primary deliverable for incident response teams, for a few reasons. Reports not only provide documented results of your efforts, but they also help you to stay focused and perform quality investigations. We use a standard template and follow reporting and language guidelines, which tends to make the reporting process consistent. Reporting forces you to slow down, document findings in a structured format, verify evidence, and think about what happened. ### Chapter 3: Pre-Incident Preparation - The policies that IR teams should be most concerned about would address expectations on the search and seizure of company-owned resources and interception of network traffic. If these two (admittedly general) issues are covered, the team should be able to perform most investigative actions. - Defining the mission of your IR team will help keep the team focused and set expectations with the rest of your organization. All elements of the team’s mission must be fully endorsed and supported by top management; otherwise, the IR team will not be able to make an impact within the organization. - During an incident, you will have several teams working concurrently: your core investigative team, ancillary teams, legal teams, and system administrators who not only respond to the core team’s tasks, but who often perform pertinent actions on their own. Good communication is paramount, and defining how that works before an incident begins is essential. - Because we work for a consulting firm, deliverables are an important part of what we provide our customers. We believe it’s important for any team to view what they do in terms of service delivery and define the standard items they will produce. The most important deliverables for an IR team are investigative reports. The reports may range from simple one-page status updates to detailed forensic examination reports that are 30 pages long each. Your IR team should explicitly define its primary deliverables, including the appropriate target completion time frames or recurrence intervals. In addition, you should create templates and instructions for each deliverable to ensure consistency. - The SysAdmin, Audit, Networking, and Security (SANS) Institute is currently the leader in the commercial IR and computer forensic training market. They have a large number of quality courses. - “Forensically Sound” Software and Admissibility of Evidence You may hear some people say that a tool must be “forensically sound,” and that its status in that regard affects the admissibility of any findings from that tool in a court of law. However, there is no strict definition of “forensically sound,” and normally a judge decides admissibility of evidence in court. We encourage your IR team to consider the factors outlined in the legal cases outlined here. In 1923, a federal court decided on a set of standards known as the Frye test. More recently, in 1993, the U.S. Supreme Court published an opinion that rewrote the standards necessary for the admissibility of scientific evidence in federal cases. (Note that the states have the freedom to adopt the standards from Frye, Dow, or their own case law.) This case, Daubert v. Merrell Dow Pharmaceuticals, 509 U.S. 579 (1993), shifted the focus from a test for general acceptance to a test for “reliability and relevance.” The judges’ findings on the admission of expert testimony resulted in the creation of a series of illustrative factors that are kept in mind during an inquiry of reliability. - Is the technique represented in a body of literature? - Understand what you have. It’s rather difficult to protect systems you don’t know exist. Your organization should consider implementing a comprehensive asset management system. In addition, performing a survey on what is actually in production, both from a system and an application perspective, can reveal details about the environment that requires additional planning. - Increase auditing to include log-on and log-off events, as well as user and group management events. • When feasible, increase auditing to include process creation and termination. • Increase local storage for each event log to a minimum of 100MB (ideally 500MB). • Forward all events to a centralized logging system. - Traffic filtering at the suborganization level  Ingress filtering and its oft-ignored counterpart, egress filtering, is absolutely essential to impede an intruder’s access across the enterprise. Consider what resources are actually required, lock the border down, and utilize change control and periodic reviews to ensure consistency over time. - If you are not in an active investigation, your regular penetration tests yield no significant results, your patch management is up to date, your IR tabletop exercises are complete, all of your logs have been reviewed, and your IR team is bored to tears, then by all means, play with honeypots. ### Chapter 4: Getting the Investigation Started on the Right Foot - For Fools rush in where Angels fear to tread. —Alexander Pope - We cover five checklists in this section: the incident summary, how the incident was detected, individual system details, network details, and malware details. There are certainly other areas we could make checklists for, but we’ve found that these are the most common and useful for an incident response investigation. Also, these checklists are not meant to be all-inclusive. You may need to add, remove, or change items to make the checklists more appropriate for your organization. We include these and other checklists from the book on our companion website, ir3e.com, as Appendix A. - Individual System Details  For each system involved, consider collecting the following information. You should avoid grouping systems together into a single document, because it’s easy to overlook details if you don’t take the time to ask these questions about each individual system. - Building an Attack Timeline In every investigation we perform, we maintain a timeline of events. Timelines keep us organized, provide context, help identify inconsistencies, and provide an overall picture of what happened. It’s important to realize that events will not necessarily be entered in chronological order. What we mean is that you will be entering events as you learn about them, not as they occur (or occurred). The following table is an abbreviated example of a timeline, sorted by the event time: - The attack timeline focuses on significant attacker activities. Record details such as when an attacker accessed a system, when files were created, when data was transferred, and when tools were executed. It is also important to record the source of the data on the timeline. For example, if you make an entry that an attacker accessed a system, include where you found that information. Finally, don’t forget to record the unique identifier of the system the event occurred on. ### Chapter 5: Initial Development of Leads - identifying the internal origin of the connection if it wasn’t available in the alert due to NAT - Most of the leads that IR teams generate consist of detectable characteristics of malicious actions. They can be represented in two types of indicators. The first type, property-based indicators, describes a set of known observable characteristics of malicious software or actions—a registry key, an MD5 hash, or a mutex with a unique name, for example. Some leads are less specific, where a combination of characteristics can define a malicious or suspicious act—unexpected executable files in the /Windows/Help directory, for example. We call these methodology-based or anomaly-based indicators. These all can be turned into indicators that one can use with single-run or enterprise-wide live response to help determine the scope of an incident. Recall that we use indicators primarily for the scoping of an incident; discover once, search everywhere. - Leads can result in host-based indicators, network-based indicators, or a combination of both. Imagine the ability to take all of the intelligence you learn from reverse-engineering a remote access trojan and rapidly tasking your network and server teams with performing searches for existing data and monitoring for future events. - Host-based indicators are the means by which we perform binary classification of an endpoint; the endpoint is either of interest in your investigation, or it is not. We create indicators by assembling a set of observables that describes a condition we know to be suspicious. Whether or not those observables result in the direct determination that a system is compromised depends on the quality of the members of that set. - A good host-based indicator is composed of a number of observables that are specific to a particular activity, yet general enough to apply to a derivative of the activity. It can be a difficult balance to achieve, particularly when the observable is based on a weak or incomplete lead. This can be best described through the following example. - Another way to improve an IOC is to describe what the binary can do. This is normally done by examining the import table. In some cases the import table is not useful—this could be due to a packer or because the author coded the binary to manually import functions. Taking a look at our sample binary, we see a large number of imports. Any individual import is not unique enough—most malware uses functions common to many other types of software. What is unique, though, is that subsets of the functions are not commonly found together in a single binary. We need to construct an IOC with multiple indistinct or loosely attributable properties. - Many times we need to create IOCs that describe what an attacker does—because there is no associated malware. Therefore, let’s build an IOC that can be used to detect a typical sequence of actions that one may observe from an attacker. These IOCs are known as methodology-based indicators and may incorporate property-based indicators with information on artifacts left behind by an active attacker. A great example of an attack whose characteristics can be modeled as a methodology-based IOC is the sethc.exe replacement attack. No malware is used, because it consists of simple registry changes or the replacement of a single file. The sethc.exe application on the Windows platform is an accessibility enhancement that helps people with various disabilities use Windows. It is invoked by pressing SHIFT five times in rapid succession and can be launched prior to a successful logon. Attackers have used this function to launch a cmd.exe session running with System privileges. - Let’s assemble the final two fragments of pseudo-code into indicators of compromise in the OpenIOC language. ### Chapter 6: Discovering the Scope of the Incident - • Will the action help answer an investigative question? • Will the action answer my questions quickly? • Am I following the evidence? • Am I putting too much effort into a single theory? • Am I using multiple independent sources of evidence? • Do I understand the level of effort? • Am I staying objective? • Am I tracking the earliest and most recent evidence of compromise? • Have I uncovered something that requires immediate remediation? ### Chapter 7: Live Data Collection - In this chapter we cover a number of topics to help you perform effective live data collections and reduce the potential risks. We explain the intent of live data collection, when it’s appropriate to perform a collection, and the data that is typically collected. We also cover how to perform a live data collection, or “live response” (LR), on common operating systems in use today. For additional good reading, see Request for Comments (RFC) 3227, “Guidelines for Evidence Collection and Archiving.” - We most frequently perform live responses, also called LRs, during an intrusion investigation; however, it may be prudent to do so during other types of investigations. You have five important factors to consider when deciding if a live response is appropriate in your current situation: 1. Is there reason to believe volatile data contains information critical to the investigation that is not present elsewhere? 2. Can the live response be run in an ideal manner, minimizing changes to the target system? 3. Is the number of affected systems large, making it infeasible to perform forensic duplications on all of them? 4. Is there risk that forensic duplications will take an excessive amount of time, or potentially fail? 5. Are there legal or other considerations that make it wise to preserve as much data as possible?