NetLog: Alternative to Fiddler and HAR captures

Have you run across where you can’t get a standard Fiddler capture, and furthermore, HAR captures from developer tools is truncating the information you need to see?

I ran across the NetLog tool built into Chromium based browsers. So, this will work in the new Microsoft Edge, Chrome, and Electron.

Here are couple known limitations before we get started…

POST request bodies are not captured.
Sites running in compatibility mode for IE sites will also not be captured.

So, with that in mind, you may need to use Fiddler or HAR capture along with using NetLog depending on the information you need to see and collect.

Let’s get started…

Optional but helpful: Close all browser tabs but one.
Navigate the tab to chrome://net-export
- For Chrome:
  Open a new tab and navigate to…
  chrome://net-export/
- For Microsoft Edge:
  Open a new tab and navigate to…
  edge://net-export
For Options, select Include raw bytes.
Leave Maximum log size blank.
In the UI that appears, press the Start Logging to Disk button.
Choose a filename to save the traffic to. Tip: Pick a location you can easily find later, like your Desktop.
Within the same Browser Window, open a new tab.
Reproduce the issue. If you close or navigate the //net-export tab, the logging will stop automatically.
After reproducing the problem, press the Stop Logging button.
Share the Net-Export-Log.json file you named from step 6.

Watch the instructions on YouTube…

Someone created a YouTube video on how to enable the logging and using Fiddler to review the capture…

Capture and diagnose network traffic from the new Chromium-based Microsoft Edge browser.

This also works on mobile devices!

This also works on Edge and Chrome for Android

This works on Chrome for iOS

You will have an email option to send the logs when on the mobile device.

For traffic in a WebView…

https://chromium.googlesource.com/chromium/src/+/HEAD/android_webview/docs/net-debugging.md

Use Fiddler to review the NetLog captures…

Install the NetLog plugin for Fiddler Classic (There is no known plugin for Fiddler Everywhere)

https://bayden.com/dl/FiddlerImportNetLog.exe

Open Fiddler
Go to File > Import Sessions
Select NetLog JSON…

More Information

For more info about NetLog…

https://dev.chromium.org/for-testers/providing-network-details

There is an online NetLog viewer (Though not as friendly as Fiddler)…

https://netlog-viewer.appspot.com/#import

Using the Online NetLog viewer, you can see additional details that Fiddler does not provide such as DNS, Timeline graph, Browser proxy config, Browser extensions installed, among other things.

Some tips and tricks with Fiddler capture

Recently, I came across a couple of scenarios where I could not get Fiddler to capture SSL traffic easily. Below are the some tips and tricks that may help in these situations.

Scenario 1: Capture Node.js web traffic in Fiddler

In the same command window where you run npm start to start the node server, run the below set commands first to set the proxy info before running npm start.

Note: The below assumes Fiddler is listening on port 8888. To find out what port Fiddler listens to, click on Tools -> options -> Connections from the Fiddler menu:

set https_proxy=http://127.0.0.1:8888
set http_proxy=http://127.0.0.1:8888
set NODE_TLS_REJECT_UNAUTHORIZED=0

Reference:

Capture network traffic in NodeJS with Fiddler – the green geek (knor.net)

Scenario 2: Capture Azure KeyVault Secret Client (from Azure SDK for .NET) with Fiddler

Set proxy info specifically in application code as below:

using Azure.Identity;
using Azure.Security.KeyVault.Secrets;

...
// add these lines in your method to call Azure Key Vault
Environment.SetEnvironmentVariable("HTTP_PROXY", "http://127.0.0.1:8888");
Environment.SetEnvironmentVariable("HTTPS_PROXY", "http://127.0.0.1:8888");

var client = new SecretClient(new Uri(keyVaultUrl), credential);
KeyVaultSecret result = client.GetSecret("MySecret");

Capturing Python web traffic with Fiddler

Update 4/15/2019 – added GraphRbacManagementClient section

Introduction:

Capturing encrypted HTTPS web traffic in Python with Fiddler can be tricky mainly because Python uses its own trusted certificate store instead of the OS’s certificate store and in certain scenario, python does not use proxy by default. This post will cover how to capture SSL traffic using Fiddler for a few different scenario:

ADAL for Python:

The problem with this case is one will get an SSL error related to certificate since Python does not trust Fiddler certificate. There are a couple of ways to resolve this:

Set the following environment variable at the beginning of the application before initializing the AuthenticationContext object

import os
…
os.environ[‘ADAL_PYTHON_SSL_NO_VERIFY‘] = “1”

Pass the verify_ssl flag to AuthenticationContext method:

# use verify_ssl=False to capture Fiddler traffic

context = adal.AuthenticationContext(authority,verify_ssl=False)

MSAL for Python:

app = msal.PublicClientApplication(client_id = appId,

authority = “https://login.microsoftonline.com/”+ tenantId,verify = False)

Python Requests Module:

The Requests module does not use Proxy by default so we have to force the request to go through Fiddler proxy. Below is an example showing how to do this.

Note: Usually Fiddler is configured to listen to port 8888. I have changed this on my system to use port 9999

import requests

…

access_token = token.get(‘accessToken’)

endpoint = ‘headers = {“Authorization”: ‘Bearer ‘ + access_token}

json_output = requests.get(endpoint,headers=headers,proxies={“http”: “http://127.0.0.1:9999“,”https”:”http:127.0.0.1:9999″},verify=False).json()

AAD Libraries for Python / GraphRbacManagementClient:

from azure.graphrbac import GraphRbacManagementClient
from azure.common.credentials import UserPassCredentials

credentials = UserPassCredentials(
<username>, # Your user name
<password>, # Your password
resource=”https://graph.windows.net”,
verify=False
)
tenant_id = <tenant name or tenant id>
graphrbac_client = GraphRbacManagementClient(credentials, tenant_id)
graphrbac_client.config.connection.verify=False
res = graphrbac_client.users.get(<UPN or ObjectID>)
print(res.display_name)

References:

https://stackoverflow.com/questions/7694789/fiddler-doesnt-capture-python-http-request

Python packet sniffing & decrypting on secure sockets with Fiddler

How to filter Fiddler capture traffic using host name and process name

This post discusses a couple of ways to filter Fiddler traffic based on domain names (or host names) and client process(es):

Note that before using filter you should make sure Fiddler is configured to capture all processes. This is indicated at the bottom left corner of Fiddler window. That area is clickable to change the selection.

Filter traffic using Fiddler’s built-in filter feature:

From Fiddler’s right pane –> Filters tab –> select “Use filters” –> under the Hosts section choose “Show only the following Hosts” –> Enter the host names you want to filter on, separated by semicolon

Note: as you edit this list the text box will change to have the yellow background to indicate the list is unsaved. Once done, you can just click on the “Actions” button to save the list and the background color should change to white.

Under the “Client Process” section, you can also select a particular process to filter on. This option is great for a standalone application. It may not be so useful for capturing browser traffic since there are multiple processes with the same name and it’s hard to tell which process is the right one to filter on.

Filter traffic using jscript code in the OnBeforeRequest function:

This option can be used especially for browser scenarios.

From Fiddler’s menu –> Rules –> “Customize Rules…” -> find the “OnBeforeRequest” function and insert the below jscript code at the beginning of the function –> Make sure to save the changes once done editting.

// begin filter
// set this to false to disable filter and true to enable filter
var filterOn: boolean = true;
if (filterOn)
{
   // hide all request by default
   oSession[“ui-hide”] = “true”;
   // here are some common processes: IE – iexplore.exe; chrome – chrome.exe, MS Edge =      MicrosoftEdgeCP.exe, IIS Express – iisexpress.exe, Powershell – powershell.exe
   // list of domain names to filter on
   var host = [“localhost”,”login.microsoftonline.com”,”graph.microsoft.com”];
   // list of processes to filter on
   var processlist = [“chrome”,”microsoftedgecp”,”iisexpress”,”powershell”];
   for (var j = 0;j < processlist.length;j++)
      {
         if (oSession.LocalProcess.Contains(processlist[j])){
            for (var i = 0;i < host.length; i++)
            {
               if(oSession.HostnameIs(host[i]))
               {
                  oSession[“ui-hide”] = null;
               }
         }
      }
   }
}
// end filter

variables:

filterOn: true to enabe filter and false to disable filter

host: contains the list of domains to filter on

processlist: contains the list of process names to filter on

References:

Modifying a Request or Response

FiddlerScript CookBook

Understanding FiddlerScript

Capture http(s) traffic with Http Fiddler

1 – Download the Fiddler 4 application and install it on the machine used to reproduce the problem (if you have not already). Go to http://www.telerik.com/download/fiddler

2 – Enable the option to decrypt HTTPS traffic: Tools -> Options -> Https -> select ‘decrypt HTTPS Traffic’ (you may be prompted to install the Fiddler certificate – make sure to select Yes)

Ensure this option is checked when collecting the trace as the data will have to be recollected if it is not.

3 – Restart the Fiddler program

(For browser-based apps)

4a. Either use private browsing mode or clear the client browser cache on the machine you will be testing from (many files are downloaded once by the browser and then cached and so will be missing from the trace unless the cache is clear; we need to see javascript and stylesheet files etc. to look for rewrite errors).

(For non browser-based apps)

4b. Launch your client application

5 – Reproduce the problem and you should see https traffic showing up in the Fiddler window.

6 – Save the resulting session output as SAZ files (File -> Save -> All Sessions)

Tracing All Network Machine Traffic Using MITMProxy for Mac OSX

Introduction

This article is meant to help you configure your Mac OS X to be able to track all your network traffic using MITMProxy. This is a free and open source alternative to Fiddler, Charles, and other network tracing alternatives for Linux/mac OS X systems. Also in addition MITMProxy is a more robust system that gives the user more configurability and programmability as there is a Python API for MITMProxy for you to integrate into your applications. There is more Documentation on MITMProxy here. Before following this documentation, please note that my OS X version may be different from yours and as OS X versions change and MITMProxy changes there may be other issues to occur. Please comment if you have any errors and I will follow up on the comments to see if I can change/add to this document to include resolutions to future errors.

This installation and configuration of MITMProxy documentation is for High Sierra:

In addition here are my versions of all dependencies, as these open source versions change some may not work properly with each other. But if you have these versions in play it should work properly :

Installing MITMProxy

The first step we will need to go through is to install MITMProxy onto your machine. You can utilize PIP to install MITMProxy, but the official documentation says to use Homebrew to install MITMProxy so we will include the documentation to install MITMProxy with Homebrew here.

In order to install Homebrew you will have to copy and paste the following command into terminal:

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

In addition to that you can learn more about Homebrew at brew.sh

After following the install script, you should now have Homebrew on your machine. Now you will be able to install MITMProxy with the command below:

brew install MITMProxy

You will probably run into an error where it says that you cannot install because the script wasn’t able to uninstall six 1.4.1.

It should look like this error:

OSError: [Errno 1] Operation not permitted: '/tmp/pip-H12_rn-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six-1.4.1-py2.7.egg-info'

This is because Six 1.4.1 is built into the OS for El Capitain, and Apple made it so that you cannot uninstall Six 1.4.1. So you may have to run this command to install MITMProxy :

brew install MITMProxy –-ignore-installed six

Now you should have MITMProxy installed onto your machine.

Configuring Mac OS High Sierra to use MITMProxy

Now that you have MITMProxy set properly, you’ll have to configure your network settings to proxy through MITMProxy. You will want to utilize the SOCKS proxy in order to get the traffic going through the machine. Currently at the current versioning of MITMProxy it utilizes SOCKS5 protocol as defined by RFC 1928 : https://tools.ietf.org/html/rfc1928

SOCKS proxy is different from most normal proxies as it is supposed to act at a much lower level as opposed to HTTPS proxying. The SOCKS protocol requires a user’s workstation to have a client installed and then will initiate a connection to the SOCKS server, then the SOCKS server will imitate the client connecting to the SOCKS server and will send packets as the client to the destination. Some VPN providers have services where they will use SOCKS5 proxy servers in order to mask the identity of the client such as changing the source IP from a European country to an American Country in order to get access to American content or vice versa. Because our Socks Proxy is actually still on the same machine it will still be the same IP Address, but we will be able to see all the requests and responses going through. Also note that VPNs are different and create a direct connection between you and the server with an encryption to keep the data even more safe. SOCKS5 Proxy sends the packets untouched, only the Source IP Address changes to the SOCKS proxy server.

So to start we are going to want to configure our machine to proxy all data through the Loopback interface utilizing SOCKS proxying in the network settings in your Mac OS X machine.

First we are going to want to go to the system preferences. You can find this by going to the top right and searching in spotlight the system preferences. After this, go to the networks setting in system preferences.

From there we are going to want to go to the advanced options and go to the Proxies tab. From there we are going to want to enable the SOCKS proxy.

After getting to the proxies tab, you will want to click the checkmark box next to SOCKS proxy, and put in 127.0.0.1:8080 for your server.

Configuring the Machine to Trust MITMProxy

After configuring your network items, press the apply button in the network panel and then your Mac should now be setup to be able to use MITMProxy, however we won’t be able to read anything going through the proxy yet because we haven’t configured the MITMproxy certs. We will need to add the certs to our keychain and then trust the certs so to tell the Mac machine that we trust MITMproxy to read all our data in between and to allow sending requests through. If you don’t trust the certificates you won’t even be able to access anything outside of your machine.

First we will want to open terminal and run mitmproxy in socks mode.

Now we have MITMProxy running on Lo0, now mitmproxy will intercept your traffic when you try to hit the endpoint mitm.it. If you see this:

This means that you are currently still accessing the internet and not going through MITMproxy, when you go through MITMproxy and try to access mitm.it you will see:

Now click on the apple and you will download a .pem file called “mitmproxy-ca-cert.pem”.

Go to downloads and put it in your keychain access. You can do this by dragging and dropping it into the keychain.

After dragging and dropping it into the keychain access, you will notice it has a red icon above the picture of the certificate. You will still need to trust the certificate in order to let data through MITMProxy. You can do this by going to trust after clicking on the certificate and then going to File > Get Info or right clicking on the item and pressing get info. Then you will have to press the carat next to trust and allow trust for the certificate.

Now you should be able to utilize MITMProxy to get data now while you’re using your applications.

You can click on the request and tab through the request, response, and detail.

You can also utilize the web browser version, I was only able to get this running in google chrome, so please be sure to get chrome beforehand. Then you can run the command:

mitmweb –mode socks5 –showhost

and then be sure to open up the localhost with the correct port number in chrome. You’ll then be able to utilize a better interface to see the response and requests.

Here is a small get request when I try go to bing.com in a separate tab and the background login starts.

Conclusion

After following all these directions we are now able to get request/responses at the socket layer using MITM. Note that as versionings change there may be issues with dependencies but with these versions you should be fine.