Ever come across a blob of JSON and wondered if it contains the information you are looking for? Have you spent time adding white space so you could see what it contains easier? Have you considered a prototype to help you review JSON data?
JSON is good for transferring data between systems, it contains the schema and the data all in one making it easy for it to be ingested by the receiving system. It is not, however, as easy to digest in this format for human users, especially if there is no whitespace. That’s where a simple prototype, like the one we’ll walkthrough below, is useful. If an analyst comes across a cyber threat report in JSON format, they can more easily digest the information if it’s in a table format. Adding sorting and filtering will make it easier for analysts to find the exact data they need in less time. Once the analyst determines the data is useful, then they can help make decisions to evaluate the rest of the data feed, or to production, the use of the data feed with their current workflow.
Let’s walk through a quick prototyping example that in the end will allow us to quickly evaluate a cyber threat report JSON bob.
Step 1 – Quick Review And Prioritization Of The Requirements
Let’s quickly consider what the customer really wants out of this prototype. They want something that will make the JSON blob easy to quickly search and digest information from. I’m going to be starting with a JSON blob, which is, by definition, JavaScript Object Notation, so let’s consider writing something using JavaScript. This way anyone with a browser can also use this prototype. It also won’t be connected to a server, which will make it even more flexible and usable.
Step 2 – Check If Something Already Exists
Now that I’ve determined that JavaScript is probably the best option here, I’ll do a quick search. Prettify has become a term used by developers who wish to print something that has usually been minimized in an easier to read format. This usually means adding whitespace, newlines, and even color to the printed text. With this knowledge, my search will be “json javascript pretty”. One of the top results is from Stackoverflow, pretty-print JSON using Javascript. Here I can see that JavaScript has a built-in function, JSON.stringify(). This is basically JavaScript’s JSON prettify function but won’t provide any sorting or filter and if the blob is more complicated with more levels or very large, it will still take time to review the JSON to search for specific data.
Step 3 – Evaluate Existing Solution
Let’s keep looking through a few of the other responses. Looks like someone has already developed a JSON report script called HTML5ReportFormat. Let’s look at see if it fits our needs. At first glance, it’s close, maybe even more than what I was originally thinking. It definitely makes digesting JSON easier and it includes sorting. Let’s take a closer look. My first issue is that the JSON blob used is statically defined. While we could replace the JSON in the script then reload the page, that doesn’t make it useable for other than developers. I want something easy for anyone to use.
Step 4 – Modify Existing Solution
Starting from the existing HTML5ReportFormat script, I’m going to enhance it a bit for usability. First, I’m going to add a textarea. This will allow users to paste a JSON blob onto the page and dynamically load a new report. After that, we’ll need a button to submit the JSON for report processing. Simple enough, now I have this:
Now the button needs to do something. This is where we must dig into the code to figure out how the report is generated. There are only a few lines of code that are not already wrapped in a function that follows the line of the statically set JSON model variable. I’ll start by wrapping those lines (323-329) in a new function so I can call it when the user clicks the new button. The function will look like:
Then I add the call to newModel() from the button I added earlier so my HTML code now looks like this:
Remember this is just a prototype, so I’m not worried about the size of my textarea or adding any fancy CSS to my button at this time. My main concern right now is making a functional prototype that meets the prioritized requirements.
Step 5 – Test
Before delivering this solution, I want to do some basic testing to ensure it will do what I tell the user it will do. I do another search for some examples using “json api” this gives me example outputs from JSON REST services. The first result happens to be from jsonapi.org. Pasting that example into my textarea works as expected. I try a few more of the various complexities and everything seems to be working as expected until I come across an API for NY Times bestsellers. Their API requires a key, which I don’t have, but they provide example outputs and that is all I need right now. Using the example here, nothing happens when I paste it into the textarea and click the Generate Report button.
Step 6 – Troubleshoot, Adjust, Test, Repeat
Time to do some quick troubleshooting. I open the browser’s developer debugger console and quickly see that JSON.parse is throwing an error. That’s a pretty easy fix, I can just wrap the code in a try/catch statement. Then to give the user some feedback, I’ll alert the user to the error. The error is somewhat helpful as it will tell you where the format error is. Might be less useful for larger unformatted JSON blobs, but this will work for now.
Now my function looks like this:
So far so good, but one of our original priority requirements was filtering. Let’s add that now. First, we’ll add an input text field and button.
Again, we’ll have to dive into the code to figure out the best place to apply a filter. There’s already a sort feature, so we will see if we can modify that. Looks like the code uses an anonymous function:
Most of that function does have to do with sorting, and I don’t see a good place to add in filtering. The last line does reference makeRows(), let’s take a look at that. Yes, this seems to be where the code is building the HTML for the table rows. We’ll be able to apply a filter in makeRows() while building each <tr>. By adding the filter and hide variables, our modified makeRows() function ends up looking like this:
If you read through the code, you’ll notice I also applied a bonus feature of highlighting the text the user filtered on. Troubleshoot, adjust, test, repeat. Filtering and highlighting seem to be working as expected.
Step 7 – Validate Solution
Now we can test using more relevant JSON blobs, such as a cyber threat report, example JSON blobs can be found here. While these JSON blobs are formatted nicely, it’s still a bit to get through the data. What we can now do is copy the JSON for the PoisionIvy report and paste it into our modified HTML5ReportFormat. Now we have a sortable list and quickly see the different types of objects included (malware, identity, campaign, attack-pattern, course-of-action, indicator, vulnerability, report, and relationship).
A quick scroll through this report also tells us this actor is taking advantage of 6 different CVEs, their actions cover the initial-compromise phase of the Mandiant-attack-lifecycle-model with a spear-phishing campaign, and the malware associated with this report is a remote access tool (RAT).
Step 8 – Deliver Prototype, Solicit Feedback
At this point, my prototype is at a good starting point. I can deliver this prototype to the customer as a first pass as it is functional and provides value. The customer can use this prototype to view the structure of the data to see if the data is providing them the information they need or expect. From here I can also start collecting some feedback and additional requirements for future iterations of the prototype.
As a developer, I can also use this prototype when evaluating APIs to ensure I’m pulling the expected data. For example this JSON of Nobel Prize winners. If I were looking for how the JSON represented non-award years for certain prizes, using the sort feature, our HTML5ReportFormat would be able to quickly show us that the Laureates field is missing, and we’ll find the information in the Overall Motivation field. This data set also shows us the sort feature doesn’t work perfectly, so I’ll add that to my future requirements list to perfect. After sorting, we get enough information to now filter on the phrase “No Nobel Prize” to get a list of just the years and categories where no award was given. We can also quickly count the number of times there has been no award (49 times across 5 different categories).
I imagine some of the feedback will include a desire for changing the background color of every other table row for easier reading, and if larger JSON blobs are being used, pagination might be a future request. Some additional features which would enhance this prototype would be to highlight where the error in the malformed JSON is and allowing users to provide a URL to grab JSON from instead of requiring them to cut/paste JSON. These additions can be done later using the evolutionary style of prototype delivery discussed in part one of this series.
Customer Benefits
Besides allowing the customer to quickly review raw JSON, other benefits in using a rapid prototyping solution, in this case, include providing a quick solution for the customer, cheaply. A prototype like this takes less than a day’s effort, maybe even only an hour or two. This prototype is also very portable, it can be used on a disconnected system, it doesn’t need access to a server, or the internet and it takes little to no spin-up time for users to start the tool.
We can also use this prototype for gathering additional requirements for a larger system. Maybe the customer has a production application that they want this functionality built into, but it takes time and a lot more engineering of the solution to get new functionality added without affecting the rest of the production application. Providing this quick solution allows the customer to get the functionality they need now and validate the need while they wait for their main tool to be updated. Then this cheap solution can be thrown away in favor of the more permanent solution.
In our three-part series on Rapid Prototyping, we looked at why rapid prototyping solutions are used and the benefits to the customer, such as meeting time-sensitive needs or addressing evolving requirements. In part two we talked about how to level-up your rapid prototyping skills by broadening your skill and tool sets. In part three we walked you through a real-world example using what we discussed in parts one and two.