Logging: The Unsung Hero in Developer Security - Here's Why and How
In the last blog we talked about what everyone assumed was the most boring topic that you could talk about, keeping your dependencies up to date. But I think I’ve got it topped this time, this time we are going to be talking about that number one thing that all developers love spending their time working on... Logging.
What does logging have to do with developer security? As it turns out a lot, in the aftermath of an incident going through the logs is how someone figures out what happened. Those working in forensics, if the logs are good enough, can provide deep insight into not only what was going on before the incident, but who was responsible for it. The initial steps that an attacker will use when they are looking through the application, fingerprinting, can look very awkward when looking at them in reverse. Every sophisticated attack starts with some very unsteady first steps.
So starting off there are a couple of things that we need to get out of the way. When we talk about logging from a cyber security stand point there are a couple of things that we really need to dig into first. Logging all the right information doesn't mean anything if they can be tampered with or removed. Because of this we need to start up front talking about what we should be doing with the logs to make them secure:
Logs should never reside on the server that generated them. If an attacker owns the server, they now also own the narrative of how they got into the server and what they did while they were there, if they leave you any logs at all. Lesson, logs don’t belong in the same place that you are running your code.
Make sure that your logs are immutable, once you have captured the information of what is going on, then you need to make sure that you have it in a form that can’t be tampered with and can be held for a long time.
How long should you keep the logs, this one depends on your threat model. Are you working in a business area where you are required to report on anything that happens for 90 days (about 3 months)? if so the minimum that you would want would be those 90 days. But here is what I will say, we know that there are often situations where customers have attackers in their environment for a long time before they are caught, the longer that you have logs the better you are going to be.
Storage for easily compressible text files is cheap if you are doing it right, depending on what you are logging and what compression you are using, you can usually get a 10:1, then ship it off to S3 Deep Glacier for around $.01 per gig.
So now that we have those items out of the way... what should go into your logs. The items we need to look at here include all the questions a reporter would want to ask (remember back in the 3rd grade?): who, what, where, why, when, and how.
Who: We need to know the person who was performing the action, who was responsible for this, if your application allows for impersonation, that should be included here as well.
What: What was the context of the action they were performing? What are they trying to do, what part of the workflow are they currently operating in.
Where: Where in the application are they coming from, what page are they making the call from, what is their IP Address (make sure that if you are using a load balancer that it’s sending the ip downstream properly).
Why: This is the toughest one to work with, but sometimes this can be as easy as indicating what part of the application they are coming from, or possibly including a session or action token.
When: When this happens, relative to both the server and the user that is using the system. If the base users time puts them in one location it really stands out when they log in from the other side of the world at 3 in the morning, those are the details that the threat hunters really love developer teams for.
How: is this call coming from html, is it coming from the API? Is it something that has come in from a 3rd party? Maybe the data is coming from an ELT process, regardless it’s important to note how they are coming in as that will help to figure out where the issue is located.
If you have all this information in your logs, there is still one major thing to deal with... One of the major things that can land you in hot water quickly is if you are logging in too much of the wrong information. The reason is that now your logs become another valuable target for the attackers giving them value. So, there are things that never go into log files, specifically PCI, Health, and Personal information.
Keeping all this information in mind can go a long way toward helping to protect your applications, as well as preventing the attackers from playing around for too long. Along with custom logs you are also going to want to make sure that you are pulling the rest of the logs as well, webserver logs – yes please, server logs - yup, database logs – you know it.
We will round out this post with the number one thing that ties us all together, it doesn’t matter if you are a developer, if you work in ops, or if you work in cybersecurity, we all benefit from developing good readable (and parsable) logs.
If you're interested in making cyber security part of your development practice and looking at it from a holistic standpoint, please do not hesitate to reach out to me. I would love to help you and your team produce safer, more secure software. As always thank you for reading our post and making your software more secure for a better world.