RUL 08.00.17 – Cybersecurity Incident Response Procedure

Authority:  Vice Chancellor for Information Technology.

History: First Issued: October 16, 2017. Last Revised: October 15, 2019.

Related Policies:
REG 08.00.02 – Computer Use Regulation

REG 08.00.03 – Data Management Procedures
RUL 08.00.14 – System and Software Security Patching Standard
RUL 08.00.16 – NC State University Security Standards for Sensitive Data and Systems

Additional References:
Appendix A – Cybersecurity Incident Response Planning Tool
Appendix B – Cybersecurity Incident Response Flowchart

NIST 800-61 – Computer Security Incident Handling Guide
CERT Coordination Center
Determining Sensitivity Levels for Shared Data
Reporting an IT Security Incident 

Contact Info:  Chief Information Security Officer, OIT 919-513-1194.

1.  Purpose

1.1. The purpose of this response plan and procedure is to detail actions required to respond effectively to an impending or active Cybersecurity Incident at NC State.

1.2. This procedure is modeled after the National Institute of Standards and Technology (NIST) Computer Security Incident Handling Guide (NIST 800-61; see Additional References section, above), which contains two key elements:

1.2.1 Taking steps to prevent Cybersecurity Incidents

1.2.2 Planning and preparing for the appropriate steps to take should an incident occur.

2.   Scope

2.1. This procedure covers the actions that must be taken in response to a Cybersecurity Incident in order to protect the confidentiality, integrity, and availability of university information systems and resources, in particular, those that could potentially impact Sensitive (purple or red) Data. This includes data and systems on campus as well as those hosted at third-party locations (for example, cloud-based services).
NOTE: This procedure depicts typical incident-response steps. However, said steps may be adjusted when specific situational conditions dictate doing so.

3.  Responsibilities

3.1. This rule applies to all system users, Data Trustees, Data Stewards, and/or Data Custodians and their delegates. NC State University REG 08.00.02 – Computer Use Regulation requires authorized users to take appropriate security precautions to protect and secure data residing in or on assigned university accounts or other university and non-university IT Resources.

3.2. The Office of Information Technology’s (OIT) Security and Compliance (S&C) unit is responsible for coordinating responses to incidents potentially involving Sensitive Data and/or posing significant risks to the University (for example, legal, financial, operational, or reputational).

3.2.1. All other incidents will be addressed at the college, division, department, or unit level with oversight and assistance from S&C, as needed.

4.  Definitions

4.1. Data Categories, Trustees, Stewards, and/or Data Custodians—Individuals who manage university and research data housed in university systems. By default, data custodians are all authorized users who access university systems and data.  Information systems development and support personnel, Local Area Network (LAN) admins/LAN-Techs, system administrators, database administrators, and third-party vendors may hold data management roles.

4.2. Dynamic Response Team (DRT)—Comprised of S&C personnel, system owners, Subject Matter Experts (SMEs), and appropriate senior administrative personnel associated with the impacted systems and data. Membership is dependent upon risk level of incident and team skills required to investigate and appropriately respond to incident.’

4.3. First Responders—College and departmental IT staff and/or system administrators who discover (or are first notified of) a potential Cybersecurity Incident.

4.4. University IT Resources—Technology assets such as computer equipment, software, networks, computer system accounts, and other digital assets and resources that primarily support the academic and administrative functions of the University. These assets are owned by the University or used by the University under contract with an external provider.  The use of these assets are governed by federal and state laws as well as University policies, regulations, and rules.

4.5. Leadership—Involved Leadership includes OIT Security & Compliance personnel and college and departmental leaders—typically including Deans, Department Heads, and Directors from impacted units. Leadership is ultimately accountable for major decisions regarding incident-response handling.

4.6. Cybersecurity Incident—Per the Computer Emergency Response Team (CERT) Coordination Center (see Additional References section, above), a security incident is an event that “violates an explicit or implied security policy,” including:

4.6.1 Attempts to gain unauthorized access to a system or data
4.6.2 Unwanted disruption, or denial of service
4.6.3 Physical damage, or theft of system or data
4.6.4 Unauthorized use of a system for processing (or storage of) data
4.6.5 Changes to system hardware, firmware, or software characteristics without the owner’s knowledge, instruction, or consent.

4.7. Sensitive Data—Any data sets with purple (ultra-highly sensitive) or red (highly sensitive) data elements; or, otherwise considered sensitive by federal or state law or contractual agreements. Examples include, but are not limited to, social security numbers, credit card numbers, Protected Health Information (PHI), Personally Identifiable Information (PII), and Controlled Unclassified Information (CUI) and other sensitive research data.  See Additional References for “Determining Sensitivity Levels for Shared Data.”

5.  Preparation

5.1. Prevention

5.1.1. According to REG 08.00.03 – Data Management Procedures, Data Stewards are required to classify data they manage; Security Administrators are responsible for ensuring that data is protected according to its sensitivity level.
5.1.2. Ongoing assessment exercises must be conducted for critical and sensitive systems to anticipate the top categories of threat susceptibility and the impact level to those systems.
5.1.3. Mitigation steps (including patching, firewalling, access controls, malware prevention, log monitoring, and user awareness and training) must be taken to reduce the likelihood of a Cybersecurity Incident. Refer to RUL 08.00.16 – NC State University Security Standards for Sensitive Data and Systems.

5.2. Planning

5.2.1. Even when following diligent prevention steps, Cybersecurity Incidents can and will occur. Therefore, incident-response planning should be conducted to outline roles and responsibilities and steps to take, should an incident occur. Appendix A, Cybersecurity Incident Response Planning Tool, shows key components to be included in each unit’s incident response (IR) plan (see Appendices).

6.   Event Reporting

6.1. Incident Discovery and Reporting

6.1.1. Cybersecurity incidents are discovered through various methods. The following are common methods for detecting compromised computers:

6.1.1.1 University Security Monitoring

  1. Intrusion Detection/Prevention System (ID/PS) alerts
  2. Vulnerability scan results
  3. Data Loss Prevention (DLP) tools
  4. Security Information and Event Management (SIEM) alerts
  5. File Integrity Monitoring (FIM) alerts
  6. External notifications

6.1.1.2  Local System Administrator Monitoring

  1. System performance degradation
  2. Anomalies detected during log monitoring
  3. Abnormal process behavior (for example, svcHost.exe executing from unexpected directory; svcHost.exe connecting over unusual ports such as port 80)

6.1.2  When any user of University IT resources discovers a potential security incident, said user must report the incident immediately using one of the following methods:

  1. External report. Follow the “Reporting an IT Security Incident” process (see Additional References, above) to report security incidents to abuse@ncsu.edu.  Incidents may be reported also by phone through the university Help Desk by calling 919-515-HELP (4357) or emailing help@ncsu.edu.
  2. Help Desk. Initial reporting on Cybersecurity Incidents may be logged via the OIT HelpDesk, as above, or be reported at college, division, department, or unit levels.
  3. Security & Compliance. S&C may discover a security incident through scanning/monitoring or other data-loss prevention software/methods.

6.1.3 First Responder Actions. To preserve evidence related to active processes running on one or more servers suspected of being compromised, first responders/system administrators may be asked to perform a memory dump and store it in a designated secured location.

IMPORTANT:  First responders/system admins must not take actions such as unplugging systems from the network, shutting down the system, and so forth, before consulting with S&C, as these actions may directly impact S&C’s ability to determine efficiently the nature and scope of incident.

6.1.4. Once a Cybersecurity Incident is discovered and reported, a ticket is created to track the status of the incident throughout the response process. Depending on level of risk involved, incidents initially reported to help desks may be escalated to S&C—including help desks from college/department/unit-level.

6.2. Initial Scoping and Investigative Analysis

6.2.1. Identification of Affected System Owners and Administrators. After the initial report has been received, the Help Desk or S&C will take steps to identify the associated system owners and administrators.

6.2.2. Verification of Investigative Level Required. S&C will work with system owners and system administrators to confirm the reported event is a Cybersecurity Incident and determine if a detailed investigative analysis is required. In addition, S&C will conduct an impact assessment with the following objectives:

  1. Ongoing Investigation. Consult with the Office of General Counsel and/or appropriate NC State entities to determine if the incident is part of an ongoing investigation into university policy or legal violations.
  2. Potential Sensitive Data Breach. Determine if Sensitive Data residing on the system is suspected of being compromised and if other connected systems could have been impacted. The following steps are performed typically in the sensitivity-level assessment:
  • Review previous Sensitive Data scans of the suspected system or connected system and/or perform a new Sensitive Data scan against the affected systems using the university-provided Sensitive Data search tools.
  • If unknown, consult with system owners, system administrators, and other key stakeholders to determine the type of data stored on or accessible from the affected systems.
  • Review the information detailed on the OIT site entitled “Determining the Sensitivity Levels of Shared Data” (see Additional References section) to identify the classification level of potentially affected data.

6.2.3. No Sensitive Data or No Cybersecurity Incident. If the classification exercise reveals that the data impacted by the event is not Sensitive Data or S&C confirms that the event was not a Cybersecurity Incident, system administrators should proceed with eradication and recovery steps to fix immediate issues, perform troubleshooting and resolution processes, and make applicable improvements to their systems as needed (see Section 9, for typical steps taken).

6.2.4. Data Is Sensitive and Cybersecurity Incident Confirmed. If the determination has been made that a Cybersecurity Incident did, in fact, occur and might have affected Sensitive Data, S&C is responsible for coordinating the remainder of the incident-response procedure, beginning with a detailed investigative analysis. S&C shall keep a written record of the incident response actions surrounding the Cybersecurity Incident.

7.  Investigative Analysis

7.1. Initial Investigative Analysis. S&C performs initial analysis, including memory dump analysis and review of other clues to develop a proper understanding of the type of Cybersecurity Incident that has occurred. The initial analysis begins during the reporting and initial discovery phase.

7.2. Dynamic Response Team. For incidents that require detailed investigative analysis, S&C works with system owners or system administrators to establish a Dynamic Response Team (DRT) for the purpose of conducting further investigation and resolving issues. The DRT may call upon other offices and resources required to carry out the investigation and remediation of the incident.

7.3. Incident Prioritization. After identifying the type of incident, the DRT performs the following steps, as appropriate:

7.3.1. Prioritize Approach by Criticality. Incident is prioritized by estimating the criticality, impact, and scope of the incident; confirming the classification of data potentially impacted; and assessing the importance level of the affected systems.
7.3.2. Perform Scope Assessment. In order to estimate the full scope of the incident, Indication of Compromise (IOC) packages may be developed to facilitate assessment against other systems connected to the compromised system.
7.3.3. Initiate Incident Response/Action Plan. The predefined departmental or unit-level incident response plan is activated or an ad hoc action plan is established.

7.4. Cyber Risk Insurance Engagement Assessment. The DRT and Leadership assess whether the nature of the incident warrants the use of cyber risk insurance and determine whether optional insurance services, such as call center and/or forensics consulting, will be needed. Insurance provider shall be notified of the incident in a manner consistent with the terms of the policy.

8.  Containment

8.1 Containment actions should be done in a methodical and delicate manner to prevent evidence tampering, tipping off attackers their actions were discovered, and preventing other implosive actions from malicious scripts (for example: malware could be configured to destroy data or render it useless should connection to attacker be interrupted).

8.1.1. Actions taken to contain the incident and prevent further damage should be based on pre-planned actions outlined in the unit’s incident-response plan. The DRT shall take appropriate actions in accordance with REG 08.00.02 – Computer Use Regulation to contain the incident and minimize risk to University.

8.1.2. Containment actions will vary based on type of system and type of incident. The following are common containment options:

  1. Temporarily disconnecting the system from the network, or using other methods to remove access to/from the affected system
  2. Blocking all communication with the system at network level
  3. Temporary removal of access to affected systems or resources
  4. Shutting down the system
  5. Disabling compromised accounts
  6. Disabling selected system functions

8.2 Forensic Investigation. As evidence is collected and preserved, stay focused on the goal to perform containment activities, to the degree possible, in parallel with additional forensics work in a manner that avoids evidence contamination (for example: a copy of the affected system may be taken for forensic work so that containment activities can begin on the original system while forensic activities are performed on the copy).

8.3 Notifications and Reporting. For incidents involving exposure or exfiltration of sensitive university data, the DRT, in collaboration with Leadership, notifies appropriate internal university stakeholders to ensure proper involvement. Any affected third-party entities are also notified as specified in the approved Data Use Agreement. The following list of stakeholders are involved typically when reporting and notification is required:

8.3.1. University Chief Information Officer (CIO)

8.3.2. University Chief Information Security Officer (CISO)

8.3.3. Incident Response Coordinator

8.3.4. Primary Forensics Examiner

8.3.5. Unit IT Security Liaison

8.3.6. System owner/administrator

8.3.7. Human Resources (for cases involving employee misconduct)

8.3.8. Office of Student Conduct (for cases involving student misconduct)

8.3.9. Appropriate Data Steward

8.3.10. Scholarships and Financial Aid (for cases involving resources used for financial aid services)

8.3.11. Office of General Counsel

8.3.12. Internal Audit

8.3.13. University Communications (for incidents that may generate publicity or require external notifications)

8.3.14. Office for Research, Innovation and Economic Development – ORIED (for incidents involving research data)

8.3.15. Insurance and Risk Management (for incidents utilizing Cyber Risk Insurance and/or HIPPA Privacy breaches)

8.3.16. Associate Vice Chancellor for Environmental Health and Public Safety and/or the University Police Department (for incidents possibly involving criminal activity)

8.3.17. Applicable third-party entity, if warranted by any Data Use Agreements
NOTE:  Reporting activities may be initiated during containment phase and extend into the eradication and recovery phase.

8.3.18. If, at any time, the DRT determines the incident involves criminal, state- or federal-mandatory reporting, or other legal issues, including misuse of state property, S&C will notify the Office of General Counsel and the University Police Department immediately (not to exceed 3 days), so that this can be reported to the appropriate authorities.

8.3.19. In the event of a security breach and exposure of Personal Information as defined by N.C.G.S. § 75-61(10), S&C will contact the Office of General Counsel immediately. Security & Compliance will notify the Office of the State Controller (OSC) within 24 hours of discovery.

8.3.20. In the event of a breach of Protected Health Information (PHI) as defined in the Health Insurance Portability and Accountability Act of 1996 and its implementing regulations (HIPAA) and other federal law, notification is required as outlined in REG 01.25.09 – Privacy/Confidentiality, Release and Security of Protected Health Information.

9.  Eradication and Recovery

9.1. After the incident has been contained, it may be necessary to implement eradication and recovery activities. System administrators shall apply the following eradication and recovery methods listed in this section, based on the nature of the incident, to eradicate infection and recover from a data exfiltration of exposure.

NOTE:  It is not unusual for eradication and recovery actions to take weeks or months, depending on the scale of the incident. Eradication and recovery steps should be prioritized appropriately and completed in a phased approach. As additional incident details are uncovered, this phase may prompt further containment and/or forensic investigation cycles.

9.2. Eradication

9.2.1. Identify and Remediate. During eradication, it is important to identify all affected hosts within the organization so they can be remediated as well. Indicators of Compromise (IOC) packages should be developed as needed (and affected networks) scanned so that all affected hosts within the organization can be addressed.
9.2.2. Account Change. It may be necessary to disable any breached accounts or change passwords of affected accounts
9.2.3. Initial Vulnerability Scans. Vulnerability scans should be performed against affected systems; any identified and exploited weaknesses must be mitigated.
9.2.4. Anti-malware. Malware scans should be performed so that malware can be identified and deleted.

9.3. Recovery

9.3.1. Rebuild or reimage. To minimize the possibility of further hidden infection, affected computers should be rebuilt or reimaged to eradicate viruses and/or other malware infection.

9.3.2. Restore. Systems can be restored from clean backups; compromised files can be restored with clean files.

9.3.3. Patching. To prevent immediate reinfection, tt is essential that systems be updated with applicable security patches, especially during eradication of infection.

9.3.4. Firewall. To prevent unauthorized connections, host-based firewalls can be enabled, network firewalls can be added, or the rules can be tightened.

9.3.5. Logging and Monitoring. Logging and monitoring should be optimized to minimize incident-detection time (for example, university file-integrity monitoring, university log monitoring, and SIEM service).

9.3.6. Overall Security Assessment. If deemed necessary, S&C may perform an overall assessment of the affected area’s security including root-cause analysis, security architecture review, tools, and procedures. S&C may then make recommendations to management for improvements.

9.3.7. Extensive Vulnerability Scans. Extensive vulnerability scans should be performed and all security weaknesses mitigated. Depending on the incident, to identify critical weaknesses in the environment, S&C may perform vulnerability scans against the impacted system (as well as all systems connected directly to them).  To prevent further breaches or exposures, system administrators are responsible for addressing identified weaknesses, including the correction of weak configuration settings and the installation of security patches, as necessary. See also RUL 08.00.14 – System and Software Security Patching Standard.

10.  Post-Incident Activity

10.1. Lessons Learned. The incident-response process should evolve to reflect new threats and technological advances. As such, ongoing Lessons Learned sessions will be coordinated by S&C to achieve closure with respect to incidents by reviewing what occurred, what was done to intervene, and how well intervention worked. Meetings will address questions such as:

10.1.1      Exactly what happened, and at what times?

10.1.2      How well did staff and management perform in dealing with the incident? Were the documented procedures followed? Were they adequate?

10.1.3      What information was needed sooner?

10.1.4      Were any steps or actions taken that might have inhibited the recovery?

10.1.5      What would the staff and management do differently the next time a similar incident occurs?

10.1.6      How could information sharing with other organizations have been improved?

10.1.7      What corrective actions can prevent similar incidents in the future?

10.1.8      What precursors or indicators should be watched for in the future to detect similar incidents?

10.1.9      Where might additional user awareness and training be needed?

10.2. Evidence Retention. The DRT, in consultation with the Office of General Counsel (OGC) and consistent with the Records Retention and Disposition Schedule, shall assess the requirements for retention of evidence gathered during investigation. Cost should be included as a key consideration.

10.3. Incidents Response Metrics. To facilitate appropriate budget and staffing plans and identify systemic security weaknesses (as well as changes in incident trends), S&C will document key metrics related to handling of incident. The following metrics are required:

10.3.1. Number of incidents

10.3.2. Man-hours per incident

10.3.3. Incident-detection time (how long after the incident occurred was it detected— relevant only for incidents investigated)

10.3.4. Likelihood of Sensitive Data exfiltration

10.3.5. Type of incident

10.3.6. Root cause

10.3.7. Estimated cost to respond and recover

10.3.8. Containment action categorization

10.3.9. Eradication and recovery categorization

11.  Regular Procedure Review

11.1. This procedure should be reviewed and updated every two years.

12.  Appendices

12.1. Appendix A – Cybersecurity Incident Response Planning Tool is a reference tool for creating an Incident Response (IR) plan. Incident Response plans shall be considered emergency-response plans and sensitive public-security information for purposes of NC Public Records laws, N.C.G.S. 132-1.6 &1.7. S&C or the DRT shall contact NC State’s University Records Officer prior to releasing any IR plans. Key components to be included in each unit’s IR plan include:

12.1.1 Contact Information—Unit/department/division/college personnel to contact in the event of a Cybersecurity Incident. This list should include both internal and external contacts (in scenarios where third parties administer your systems). In addition, central S&C contacts should be known. All contacts playing a part in the incident-response process must clearly understand their roles and responsibilities.

12.1.2 Incident Reporting Mechanism—Clear documentation of methods to be used to report incidents including email addresses, help desk phone numbers, on-call contacts, and so forth.

12.1.3 Critical/Sensitive System—Identification of critical or sensitive systems (which store, process, or transmit Sensitive Data) should be identified on a network diagram—including pertinent information about interconnected systems and the details involved for each connection.

12.1.4 Potential Attack Vectors—To conceptualize containment, eradication, and recovery strategies, identify the most likely ways the given system could be attacked.

12.1.5 Containment, Eradication and Recovery—For each potential attack vector, develop a strategy for how to contain the attack, how to eradicate the infection, and a develop detailed procedure for how to recover from the attack.

12.1.6 Secure Storage Location—During the incident, especially for first responders, identify a secure storage location for evidence gathered (for example, memory dumps, system images, and so forth). In addition, clear instructions should be made available for accessing and using first responder incident-response tools.

12.1.7 Common Indications of Compromise—To receive warnings or notifications of potential compromises, system administrators, system owners, and other key stakeholders may be required to subscribe to university tools (for example, SIEM, FIM, Vulnerability Scans, and so forth).

12.2. Appendix B – Cybersecurity Incident Response Flowchart shows the typical response phases when a Cybersecurity Incident is discovered, as well as roles and responsibilities for those involved in responding to the incident.