Understanding Data Sources and File Formats

Windows Incident Response

Following on the heels of my previous post regarding file formats and sharing the link to the post on LinkedIn, I had some additional thoughts that would benefit greatly from not blasting those thoughts out as comments to the original post, but instead editing and refining them via this medium.

My first thought was, is it necessary for every analyst to have deep, intimate knowledge of file formats? The answer to that is a resounding “no”, because it’s simply not possible, and not scalable. There are too many possible file formats for analysts to be familiar with; however, if a few knowledgeable analysts, ones who understand the value of the file format information to DFIR, CTI, etc., document the information and are available to act as resources, then that should suffice. With the format and it’s value documented and reviewed/updated regularly, this should act as a powerful resource.

What is necessary is that analysts be familiar enough with data sources and file formats to understand when something is amiss, or when something is not presented by the tools, and know enough recognize that fact. From there, simple troubleshooting steps can allow the analyst to develop a thoughtful, reasoned question, and to seek guidance. 

So, why is understanding file formats important for DFIR analysts?

1. Parsing
First, you need to understand how to effectively parse the data. Again, not every analyst needs to be an expert in all file formats – that’s simply impossible. But, if you’re working on Windows systems, understanding file formats such as the MFT and USN change journal, and how they can be tied together, is important. In fact, it can be critical to correctly answering analysis questions. Many analysts parse these two file separately, but David and Matt’s TriForce tool allowed these files (and the $LogFile) to be automatically correlated.

So, do you need to be able to write an OLE parser when so many others already exist? No, not at all. However, we should have enough of an understanding to know that certain tools allow us to parse certain types of files, albeit only to a certain level. We should a

[…]
Content was cut in order to protect the source.Please visit the source for the rest of the article.