Parsing web logs and sharpening my C#

It’s been too long

I dived back into C# this past week by revisiting one of my old projects – the web log viewer that I used to review the server logs for my website. It had previously been done in the .NET Framework 4.something and I wanted a new version that would handle a month’s worth of the GZip file downloads available from my hosting service and maybe output the merged file to a new file.

There’s a separate log file for each day so I needed to be able to load individual files or an entire directory’s worth and I wanted it to save me the hassle of manually decompressing each file. The previous version handled ZIP format so I decided to have this one handle both for flexibility.

It had been more than a minute before I even cracked open Visual Studio, probably since last July when I did an update of my mortgage calculator project on Github, as I’d been focusing on other things for awhile and it’s easy to let time slip away.

The main work on the program is done although I still need it to export the data to a new file. The code for the project is available for download on Github.

What I learned this week

1.) Don’t put the tools away for too long.

When I was teaching C# and SQL Server years ago, I told my students that programming is like any other skill – if you don’t use it, you lose it. Of course, once you have a solid foundation, it’s easier to pick it back up but it’s still as painful in its own way as that first week back to the gym. I even used w3Schools C# exercises as a refresher on a wide range of topics.

2.) DataTables and DataGrids are memory hogs.

This might be unfair since the standard Windows Forms DataGrid was probably not meant to handle the 1.2 million entries from my March web logs. Still, when imported into the program, a few hundred megabytes of log data caused the program to occupy 2.1 gigabytes of memory.

I tried a refactoring to get rid of the DataTables after ChatGPT pointed out that they are inherently expensive for memory usage which provided no improvement. On GPT’s suggestion, I even abandoned using the DataSource behind the grid and tried adding rows manually. This was horribly slow and even more expensive to which GPT basically said “Oh yeah, that’s the worst thing you could do.”

There might be other options such as loading the entire collection into a database in the background and then pulling selected rows but, for now, I’ll just be glad for my 16 GB of RAM.

3.) Updating a form’s toolbar from a custom class

Most of the log processing functions are in a static class called LogProcess and I wanted to send a message back to the main form for each file that was processed from a directory so the program wouldn’t just seem to hang while loading 30 files. This turned out to be relatively easy although selecting between the IProcess<T> and classic events method was a little more difficult. IProcess<T> has less code but the classic events model just worked better with this app which chooses between two class functions depending on whether it’s processing a file or a directory.

Even classic events seem a little simpler than they used to be.

Declare the event in the custom class.

public static event Action FileOp;

Invoke the event at the appropriate time.

FileOp?.Invoke("Processing " + file);

Add async to the event declaration on the form.

    private async void cmdLoad_Click(object sender, EventArgs e)

Define event in the form’s click event.

            LogProcess.FileOp += (value) =>
            {
                if (InvokeRequired)
                    Invoke(new Action(() => tssStatus.Text = value));
                else
                    tssStatus.Text = value;
            };

Then invoke the event at the proper time –

FileOp?.Invoke("Processing " + file);

Another problem with the IProcess<T> was that it wasn’t handing control back to my form’s click event so that I could further update the task bar after the class was finished. Still, I need to hold onto it to try in future projects.

4.) .NET handles ZIP and GZ natively, even if it might be limited.

The last time I needed .NET to work with ZIP files, I was working with Ionic’s DotNetZip which is now deprecated. Fortunately, .NET handles both formats just fine for what I needed and the code is simple enough.

You can see the full code in the LogProcess.DecompressFile function available in the Github repository.

5.) My restraint on A.I. use is getting more nuanced.

I spent a lot of time going back and forth with ChatGPT over the last few days, asking questions and having it review code for errors. I can’t ignore its value as a knowledge repository and analysis tool. The one thing I do not ask it to do is generate code for me. When it provides example code, I break it down and make sure I understand what it’s doing and then, if I use that method, I re-write it myself.

The vibe coders out there will probably laugh at this as a waste of time as if productivity and convenience are the only considerations but, for me, the extra work is where I find the real value in programming. I could have the A.I. generate all the code and keep copying and pasting until I just get something that works but what would I learn from that except how to depend on an A.I. tool?

The project was difficult and frustrating at times – both my C# and my algorithmic thinking skills were a little rusty – but that difficulty and my willingness to push through it are what allowed me to learn as much as I did and that knowledge is now mine. I also took the time to ask the questions about memory usage and UI issues and earned the anwers through investigation. I let A.I. be my assistant, not my replacement.

Still, a lot of questions I directed to ChatGPT reminded me of the one engineering job where we had to be told to investigate issues on our own before asking the one database admin in the organization whose experience and increased access would enable him to solve anything for us. Going to that guy came to be known as “pressing the Bob button”, synonymous with taking the easy way out. We tried to avoid it because it meant Bob was distracted from the rest of his work and it detracted from our value as engineers.

I’d be foolish to avoid A.I. completely but I think it’s important that I continue to be vigilant in my use of it and always ask “Is this something I would benefit from figuring out myself with just a little more effort?”.