Jekyll2021-01-24T09:30:00+00:00/feed.xmlBad GatewaySharing thoughts about apps development and Kanye WestLucas LegnameUsing mitmproxy through Docker with a M1 Mac2021-01-22T08:20:20+00:002021-01-22T08:20:20+00:00/mitmproxy/2021/01/22/mitmproxy-m1-mac<p>The new M1-powered Macs are available since a few months now and even though most apps are already compatible with the ARM chip (natively or through <strong>Rosetta 2</strong>), there is still one big missing : <strong>Docker</strong>.</p>
<p>To be accurate, Docker is currently available on M1 Macs but only as a <strong>Tech Preview</strong>. If you are looking for something stable, you should better wait a bit but if you don’t mind wasting some time on weird bugs and want to experience Docker on ARM before everyone else then this build is for you.</p>
<p>In a previous post, I suggested using <strong>mitmproxy</strong> through <strong>Docker</strong> to keep you computer clean and ease versions switching. So as I freshly received my M1-powered Macbook Air, I wanted to do the same thing with my new computer.</p>
<p><img src="/assets/images/docker/logo.png" alt="Docker logo" /></p>
<h2 id="lets-run-the-official-mitmproxy-docker-image">Let’s run the official mitmproxy Docker image</h2>
<p>Once Docker was set up and running, I naturally tried to use the official <strong>mitmproxy</strong> Docker image as I used to do on my previous Intel-powered Mac.</p>
<p>So I simply ran the following command :</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker run <span class="nt">--rm</span> <span class="nt">-it</span> <span class="nt">-p</span> 8080:8080 mitmproxy/mitmproxy:6.0.2
</code></pre></div></div>
<p>And… it mostly worked ! I was able to use the official <strong>mitmproxy</strong> image but I still got this annoying warning message on start :</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>WARNING: The requested image<span class="s1">'s platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
</span></code></pre></div></div>
<p>This message appears because <strong>mitmproxy</strong> does not offer multi platform images for the moment. If you check the tags from mitmproxy’s <a href="https://hub.docker.com/r/mitmproxy/mitmproxy/tags" target="_blank">DockerHub registry</a>, you will notice that there is only one architecture (<code class="language-plaintext highlighter-rouge">linux/amd64</code>) per tag.</p>
<p>We can still run containers using this image thanks to <a href="https://www.docker.com/blog/multi-arch-images/" target="_blank">Docker’s QEMU emulator</a> but if we want to avoid the previous warning message and enjoy better performance, we need to build a multi platform <strong>Docker</strong> image.</p>
<p><img src="/assets/images/docker/docker-arch.png" alt="mitmproxy is alive" /></p>
<h2 id="building-a-multi-platform-docker-image">Building a multi platform Docker image</h2>
<p>As the <strong>mitmproxy</strong> project source code is <a href="https://github.com/mitmproxy/mitmproxy" target="_blank">available on Github</a>, I wanted to try building my own ARM-friendly Docker image. So let’s checkout the code and see how it goes.</p>
<p>After a few researches, I found out I would need to use Docker <strong>Buildx</strong> to create a multi platform image. This tool is an experimental feature so if you intend to use it, <strong>make sure experimental features are enabled</strong> in your Docker settings.</p>
<p>Using <strong>Buildx</strong> is quite simple, first we need to create our own builder :</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker buildx create <span class="nt">--name</span> mybuilder
docker buildx use mybuilder
docker buildx inspect <span class="nt">--bootstrap</span>
</code></pre></div></div>
<p>Once the builder is created the rest is straightforward : open the terminal, change directory to the root <code class="language-plaintext highlighter-rouge">mitmproxy</code> folder and execute the following command.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker buildx build <span class="nt">--push</span> <span class="nt">-t</span> lucaslegname/mitmproxy:6.0.2 <span class="nt">--platform</span> linux/amd64,linux/arm64 <span class="nt">--file</span> release/docker/Dockerfile <span class="nb">.</span>
</code></pre></div></div>
<p>This command will build a Docker image compatible with the <code class="language-plaintext highlighter-rouge">linux/amd64</code> and <code class="language-plaintext highlighter-rouge">linux/arm64</code> architectures and push it to <a href="https://hub.docker.com/r/lucaslegname/mitmproxy" target="_blank">a DockerHub registry</a>.</p>
<p>Now you can run this new M1-friendly image using the following command :</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker run <span class="nt">--rm</span> <span class="nt">-it</span> <span class="nt">-p</span> 8080:8080 lucaslegname/mitmproxy:6.0.2
</code></pre></div></div>
<p>That’s it ! You can now run <strong>mitmproxy</strong> through <strong>Docker</strong> with an optimised image and you even know how to create multi platform images using <strong>Buildx</strong>. Enjoy !</p>
<p><img src="/assets/images/mitmproxy/mitmproxy-empty.jpg" alt="mitmproxy is alive" /></p>
<h3 id="sidenotes">Sidenotes</h3>
<ul>
<li>
<p>by default, M1 Macs build <code class="language-plaintext highlighter-rouge">linux/arm64</code> images so you can also use <strong>Buildx</strong> in case you need to share <strong>Docker</strong> images with people <del>still</del> using Intel-powered computers</p>
</li>
<li>
<p><strong>Buildx</strong> can also be very handy to build <strong>Raspberry-Pi-compatible images</strong> from an Intel-powered computer</p>
</li>
</ul>
<h2 id="references">References</h2>
<ul>
<li><a href="https://www.docker.com/blog/multi-arch-build-and-images-the-simple-way/" target="_blank">Multi-arch build and images, the simple way</a></li>
<li><a href="https://medium.com/better-programming/how-to-actually-deploy-docker-images-built-on-a-m1-macs-with-apple-silicon-a35e39318e97" target="_blank">How to Actually Deploy Docker Images Built on M1 Macs With Apple Silicon</a></li>
<li><a href="https://www.docker.com/blog/multi-arch-images/" target="_blank">Building Multi-Arch Images for Arm and x86 with Docker Desktop</a></li>
</ul>Lucas LegnameIn a previous post, I suggested using mitmproxy through Docker. So as I freshly received my M1-powered Macbook Air, I wanted to do the same thing with my new computer.Creating scripts for mitmproxy2020-11-04T18:20:20+00:002020-11-04T18:20:20+00:00/mitmproxy/2020/11/04/mitmproxy-scripts<p>Now that we covered the first steps to get started with <strong>mitmproxy</strong> (<a href="/mitmproxy/2020/04/10/mitmproxy.html">here</a> and <a href="/mitmproxy/2020/04/22/mitmproxy-tips.html">here</a>), I would like to develop a bit more one of the best features the tool has to offer : the ability to run <strong>Python</strong> scripts.</p>
<p>While writting addons is purely optional, automating requests (or responses) editing with custom scripts considerably extends the boundaries of mitmproxy. Through this post, we will see a few concrete examples of how you can use addons to test apps.</p>
<p>If you already managed to set everything in order to run the program and catch requests, writting scripts should not be too complicated for you. Feel free to re-use and alter the examples provided in this post in order to create your own scripts.</p>
<p><img src="/assets/images/mitmproxy/script-02.png" alt="Script excerpt" /></p>
<h2 id="writting-and-executing-scripts">Writting and executing scripts</h2>
<p>As I mentioned earlier, scripts for mitmproxy must be written using the <a href="https://en.wikipedia.org/wiki/Python_(programming_language)" target="_blank">Python</a> programming language. The Python syntax is not particularly complicated and if you already used any object-oriented programming language then you should be fine.</p>
<p>One advice though : <strong>pay attention to indentation</strong>! Python is an indent-sensitive language which means that it uses leading whitespaces (instead of curly-brackets <code class="language-plaintext highlighter-rouge">{}</code>) in order to compute the indentation level of the line.</p>
<h3 id="example">Example</h3>
<p>As the best way to learn is often to see an example, here is a simple one. This script will change the HTTP code of any response which’s url matches the <code class="language-plaintext highlighter-rouge">filter</code> param.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">mitmproxy</span> <span class="kn">import</span> <span class="n">http</span>
<span class="kn">from</span> <span class="nn">mitmproxy</span> <span class="kn">import</span> <span class="n">ctx</span>
<span class="k">class</span> <span class="nc">ChangeHTTPCode</span><span class="p">:</span>
<span class="nb">filter</span> <span class="o">=</span> <span class="s">"netflix.com"</span>
<span class="k">def</span> <span class="nf">response</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">flow</span><span class="p">:</span> <span class="n">http</span><span class="p">.</span><span class="n">HTTPFlow</span><span class="p">)</span> <span class="o">-></span> <span class="bp">None</span><span class="p">:</span>
<span class="k">if</span> <span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="nb">filter</span> <span class="ow">in</span> <span class="n">flow</span><span class="p">.</span><span class="n">request</span><span class="p">.</span><span class="n">pretty_url</span><span class="p">):</span>
<span class="n">flow</span><span class="p">.</span><span class="n">response</span><span class="p">.</span><span class="n">status_code</span> <span class="o">=</span> <span class="mi">503</span>
<span class="n">addons</span> <span class="o">=</span> <span class="p">[</span><span class="n">ChangeHTTPCode</span><span class="p">()]</span>
</code></pre></div></div>
<p>To execute the addon, just save it in a file named <code class="language-plaintext highlighter-rouge">myscript.py</code> and execute the following command :</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mitmproxy <span class="nt">-s</span> ./myscript.py
</code></pre></div></div>
<p>Now if you try to start the <strong>Netflix</strong> app with a device set to use mitmproxy, all requests will appear as errors : well done, you just forced the app to show its error screen!</p>
<p><img src="/assets/images/mitmproxy/script-01.png" alt="Netflix script" /></p>
<h3 id="events">Events</h3>
<p>In the previous example, we used the <code class="language-plaintext highlighter-rouge">response</code> event in order to alter the response before it reaches the device. <strong>mitmproxy</strong> allows developers to access to multiple <strong>HTTP</strong>, <strong>TCP</strong>, <strong>WebSocket</strong> or <strong>generic</strong> events.</p>
<p>The full list of supported events is available in the <a href="https://docs.mitmproxy.org/stable/addons-events/" target="_blank">mitmproxy documentation</a> but let’s have a look at several handy ones.</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">load()</code> is triggered <strong>when you launch mitmproxy</strong>, it is very convenient if you need to adapt the tool’s settings to run your script</li>
</ul>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">load</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">loader</span><span class="p">):</span>
<span class="n">ctx</span><span class="p">.</span><span class="n">options</span><span class="p">.</span><span class="n">http2</span> <span class="o">=</span> <span class="bp">False</span>
</code></pre></div></div>
<ul>
<li><code class="language-plaintext highlighter-rouge">request()</code> is triggered <strong>before the request is sent</strong>, the perfect moment to override an <strong>user-token</strong> for example</li>
</ul>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">request</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">flow</span><span class="p">:</span> <span class="n">http</span><span class="p">.</span><span class="n">HTTPFlow</span><span class="p">)</span> <span class="o">-></span> <span class="bp">None</span><span class="p">:</span>
<span class="n">flow</span><span class="p">.</span><span class="n">request</span><span class="p">.</span><span class="n">headers</span><span class="p">[</span><span class="s">"authorization"</span><span class="p">]</span> <span class="o">=</span> <span class="s">"Bearer NEWUSERTOKEN"</span>
</code></pre></div></div>
<ul>
<li><code class="language-plaintext highlighter-rouge">response()</code> is triggered <strong>before the response reaches the device</strong>, the right moment to override the <strong>response content</strong> for example</li>
</ul>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">response</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">flow</span><span class="p">:</span> <span class="n">http</span><span class="p">.</span><span class="n">HTTPFlow</span><span class="p">)</span> <span class="o">-></span> <span class="bp">None</span><span class="p">:</span>
<span class="n">flow</span><span class="p">.</span><span class="n">response</span><span class="p">.</span><span class="n">content</span> <span class="o">=</span> <span class="s">"custom response content"</span>
</code></pre></div></div>
<h3 id="the-request-and-response-objects">The request and response objects</h3>
<p>Both <strong>request</strong> and <strong>response</strong> objects are part of the <code class="language-plaintext highlighter-rouge">http.HTTPFlow</code> variable passed as a parameter to all HTTP events.</p>
<p>Editing the request (or response) is quite simple as long as you have hints about the object structure : my best advice here is to directly have a look at the project <a href="https://github.com/mitmproxy/mitmproxy" target="_blank">Github repository</a> and check the classes.</p>
<ul>
<li>Here are a few essential attributes from the <a href="https://github.com/mitmproxy/mitmproxy/blob/v5.1.1/mitmproxy/net/http/request.py#L50" target="_blank">request object</a> :</li>
</ul>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="bp">self</span><span class="p">.</span><span class="n">method</span> <span class="o">=</span> <span class="n">method</span>
<span class="bp">self</span><span class="p">.</span><span class="n">host</span> <span class="o">=</span> <span class="n">host</span>
<span class="bp">self</span><span class="p">.</span><span class="n">port</span> <span class="o">=</span> <span class="n">port</span>
<span class="bp">self</span><span class="p">.</span><span class="n">path</span> <span class="o">=</span> <span class="n">path</span>
<span class="bp">self</span><span class="p">.</span><span class="n">headers</span> <span class="o">=</span> <span class="n">headers</span>
<span class="bp">self</span><span class="p">.</span><span class="n">content</span> <span class="o">=</span> <span class="n">content</span>
</code></pre></div></div>
<p>💡 <em>It is also possible to use the <code class="language-plaintext highlighter-rouge">self.url</code> property/setter to update the full url</em></p>
<ul>
<li>Here are a few essential attributes from the <a href="https://github.com/mitmproxy/mitmproxy/blob/v5.1.1/mitmproxy/net/http/response.py#L37" target="_blank">response object</a> :</li>
</ul>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="bp">self</span><span class="p">.</span><span class="n">status_code</span> <span class="o">=</span> <span class="n">status_code</span>
<span class="bp">self</span><span class="p">.</span><span class="n">headers</span> <span class="o">=</span> <span class="n">headers</span>
<span class="bp">self</span><span class="p">.</span><span class="n">content</span> <span class="o">=</span> <span class="n">content</span>
</code></pre></div></div>
<p>💡 <em>You can also use the <code class="language-plaintext highlighter-rouge">self.reason</code> property to get the reason phrase corresponding to the status code (ex : “Not found” for a 404 error)</em></p>
<h3 id="sending-logs">Sending logs</h3>
<p>The console can be very useful when you are developing new scripts : to open the console, hit <strong>E</strong> from the main screen.</p>
<p>To send logs from a script, you can use the following function :</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">ctx</span><span class="p">.</span><span class="n">log</span><span class="p">.</span><span class="n">info</span><span class="p">(</span><span class="s">"Hello world"</span><span class="p">)</span>
</code></pre></div></div>
<p>Make sure to use a level of verbosity (<code class="language-plaintext highlighter-rouge">debug</code>,<code class="language-plaintext highlighter-rouge">alert</code>, <code class="language-plaintext highlighter-rouge">info</code>, <code class="language-plaintext highlighter-rouge">warn</code> and <code class="language-plaintext highlighter-rouge">error</code>) high-enough to see you logs. <strong>mitmproxy</strong>’s default level of verbosity is <code class="language-plaintext highlighter-rouge">info</code>.</p>
<h2 id="using-scripts-to-debug-apps">Using scripts to debug apps</h2>
<p>Now that we have seen the main events and objects structure, let’s see a few concrete examples where you can use scripts to debug apps.</p>
<h3 id="redirecting-requests-to-another-environment">Redirecting requests to another environment</h3>
<p>This first case is pretty common : you have an app that gets data from an API. The next version of your API is available on the <strong>preprod</strong> environment and you would like to test it using the exact same build that the one available on Google Play (or on the Appstore).</p>
<p>Of course, the store build of your app is configured to send requests to your <strong>prod</strong> env and no debug setting is accessible.</p>
<p>Here, we will intercept requests intended to the <strong>prod</strong> environment and re-route them to the <strong>preprod</strong> server using the following script :</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">mitmproxy</span> <span class="kn">import</span> <span class="n">http</span>
<span class="kn">from</span> <span class="nn">mitmproxy</span> <span class="kn">import</span> <span class="n">ctx</span>
<span class="k">class</span> <span class="nc">Redirect</span><span class="p">:</span>
<span class="n">redirect_rules</span> <span class="o">=</span> <span class="p">{</span>
<span class="s">"api.domain.com"</span><span class="p">:</span> <span class="s">"api-preprod.domain.com"</span>
<span class="p">}</span>
<span class="k">def</span> <span class="nf">load</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">loader</span><span class="p">):</span>
<span class="n">ctx</span><span class="p">.</span><span class="n">options</span><span class="p">.</span><span class="n">http2</span> <span class="o">=</span> <span class="bp">False</span> <span class="c1"># HTTP2 won't let you change the url
</span>
<span class="k">def</span> <span class="nf">request</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">flow</span><span class="p">:</span> <span class="n">http</span><span class="p">.</span><span class="n">HTTPFlow</span><span class="p">)</span> <span class="o">-></span> <span class="bp">None</span><span class="p">:</span>
<span class="k">for</span> <span class="n">init_domain</span><span class="p">,</span> <span class="n">new_domain</span> <span class="ow">in</span> <span class="bp">self</span><span class="p">.</span><span class="n">redirect_rules</span><span class="p">.</span><span class="n">items</span><span class="p">():</span>
<span class="k">if</span> <span class="p">(</span><span class="n">init_domain</span> <span class="ow">in</span> <span class="n">flow</span><span class="p">.</span><span class="n">request</span><span class="p">.</span><span class="n">pretty_url</span><span class="p">):</span>
<span class="n">flow</span><span class="p">.</span><span class="n">request</span><span class="p">.</span><span class="n">url</span> <span class="o">=</span> <span class="n">flow</span><span class="p">.</span><span class="n">request</span><span class="p">.</span><span class="n">pretty_url</span><span class="p">.</span><span class="n">replace</span><span class="p">(</span><span class="n">init_domain</span><span class="p">,</span> <span class="n">new_domain</span><span class="p">)</span>
<span class="n">addons</span> <span class="o">=</span> <span class="p">[</span><span class="n">Redirect</span><span class="p">()]</span>
</code></pre></div></div>
<p>About this example :</p>
<ul>
<li>you can add multiple redirect rules in the <code class="language-plaintext highlighter-rouge">redirect_rules</code> object</li>
<li>we use a simple “replace” function to update the url so you can replace the domain or any other part of the string</li>
<li>since you cannot change requests url when using HTTP/2, we force the use of HTTP on launch</li>
</ul>
<h3 id="use-a-special-user-agent-for-testing">Use a special user-agent for testing</h3>
<p>Now imagine that you are debugging an API and that you need to easily find the logs produced by your device among all production logs.</p>
<p>The simplest way to achieve this is to use a custom <strong>user-agent</strong>. Fortunately, doing this is straightforward with the following script :</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">mitmproxy</span> <span class="kn">import</span> <span class="n">http</span>
<span class="kn">from</span> <span class="nn">mitmproxy</span> <span class="kn">import</span> <span class="n">ctx</span>
<span class="k">class</span> <span class="nc">ChangeUserAgent</span><span class="p">:</span>
<span class="n">user_agent</span> <span class="o">=</span> <span class="s">"my-custom-user-agent"</span>
<span class="k">def</span> <span class="nf">request</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">flow</span><span class="p">:</span> <span class="n">http</span><span class="p">.</span><span class="n">HTTPFlow</span><span class="p">)</span> <span class="o">-></span> <span class="bp">None</span><span class="p">:</span>
<span class="n">flow</span><span class="p">.</span><span class="n">request</span><span class="p">.</span><span class="n">headers</span><span class="p">[</span><span class="s">"user-agent"</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="p">.</span><span class="n">user_agent</span>
<span class="n">addons</span> <span class="o">=</span> <span class="p">[</span><span class="n">ChangeUserAgent</span><span class="p">()]</span>
</code></pre></div></div>
<p>About this example :</p>
<ul>
<li>using the <code class="language-plaintext highlighter-rouge">flow.request.headers</code> array you can update any header value (<code class="language-plaintext highlighter-rouge">authorization</code>, <code class="language-plaintext highlighter-rouge">accept-language</code>, …)</li>
</ul>
<h3 id="create-an-editable-cache-for-api-responses">Create an editable cache for API responses</h3>
<p>For this last example, imagine you need to test how an app behaves when it receives a specific response after sending a request.</p>
<p>Wouldn’t it be great to be able to save requests locally in order to edit them manually ? That is exactly what the following script allows you to do.</p>
<p>The code from this one is far from being perfect but it does the job quite well :</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">mitmproxy</span> <span class="kn">import</span> <span class="n">http</span>
<span class="kn">from</span> <span class="nn">mitmproxy</span> <span class="kn">import</span> <span class="n">ctx</span>
<span class="kn">import</span> <span class="nn">os</span>
<span class="k">class</span> <span class="nc">EditableCache</span><span class="p">:</span>
<span class="n">url_filter</span> <span class="o">=</span> <span class="s">"www.only-redirect-requests-from-this-domain.com"</span>
<span class="n">path</span> <span class="o">=</span> <span class="s">"/Full/path/to/cache/folder"</span>
<span class="k">def</span> <span class="nf">response</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">flow</span><span class="p">:</span> <span class="n">http</span><span class="p">.</span><span class="n">HTTPFlow</span><span class="p">)</span> <span class="o">-></span> <span class="bp">None</span><span class="p">:</span>
<span class="k">if</span> <span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">url_filter</span> <span class="ow">in</span> <span class="n">flow</span><span class="p">.</span><span class="n">request</span><span class="p">.</span><span class="n">pretty_url</span><span class="p">):</span>
<span class="n">fullpath</span> <span class="o">=</span> <span class="bp">self</span><span class="p">.</span><span class="n">path</span> <span class="o">+</span> <span class="s">"/"</span> <span class="o">+</span> <span class="bp">self</span><span class="p">.</span><span class="n">clean_url</span><span class="p">(</span><span class="n">flow</span><span class="p">.</span><span class="n">request</span><span class="p">.</span><span class="n">url</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">isfile</span><span class="p">(</span><span class="n">fullpath</span><span class="p">):</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">exists</span><span class="p">(</span><span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">dirname</span><span class="p">(</span><span class="n">fullpath</span><span class="p">)):</span>
<span class="n">os</span><span class="p">.</span><span class="n">makedirs</span><span class="p">(</span><span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">dirname</span><span class="p">(</span><span class="n">fullpath</span><span class="p">))</span>
<span class="nb">file</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="n">fullpath</span><span class="p">,</span> <span class="s">"w"</span><span class="p">)</span>
<span class="nb">file</span><span class="p">.</span><span class="n">write</span><span class="p">(</span><span class="n">flow</span><span class="p">.</span><span class="n">response</span><span class="p">.</span><span class="n">text</span><span class="p">)</span>
<span class="nb">file</span><span class="p">.</span><span class="n">close</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">request</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">flow</span><span class="p">:</span> <span class="n">http</span><span class="p">.</span><span class="n">HTTPFlow</span><span class="p">)</span> <span class="o">-></span> <span class="bp">None</span><span class="p">:</span>
<span class="k">if</span> <span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">url_filter</span> <span class="ow">in</span> <span class="n">flow</span><span class="p">.</span><span class="n">request</span><span class="p">.</span><span class="n">pretty_url</span><span class="p">):</span>
<span class="n">fullpath</span> <span class="o">=</span> <span class="bp">self</span><span class="p">.</span><span class="n">path</span> <span class="o">+</span> <span class="s">"/"</span> <span class="o">+</span> <span class="bp">self</span><span class="p">.</span><span class="n">clean_url</span><span class="p">(</span><span class="n">flow</span><span class="p">.</span><span class="n">request</span><span class="p">.</span><span class="n">url</span><span class="p">)</span>
<span class="k">if</span> <span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">isfile</span><span class="p">(</span><span class="n">fullpath</span><span class="p">):</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">fullpath</span><span class="p">,</span> <span class="s">'r'</span><span class="p">)</span> <span class="k">as</span> <span class="n">cache_file</span><span class="p">:</span>
<span class="n">cache_content</span> <span class="o">=</span> <span class="n">cache_file</span><span class="p">.</span><span class="n">read</span><span class="p">()</span>
<span class="n">flow</span><span class="p">.</span><span class="n">response</span> <span class="o">=</span> <span class="n">http</span><span class="p">.</span><span class="n">HTTPResponse</span><span class="p">.</span><span class="n">make</span><span class="p">(</span><span class="mi">200</span><span class="p">,</span><span class="n">cache_content</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">clean_url</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">url</span><span class="p">):</span>
<span class="n">filename</span> <span class="o">=</span> <span class="n">url</span>
<span class="n">filename</span> <span class="o">=</span> <span class="n">filename</span><span class="p">.</span><span class="n">replace</span><span class="p">(</span><span class="s">"https://"</span><span class="p">,</span> <span class="s">""</span><span class="p">)</span>
<span class="n">filename</span> <span class="o">=</span> <span class="n">filename</span><span class="p">.</span><span class="n">replace</span><span class="p">(</span><span class="s">"?"</span><span class="p">,</span> <span class="s">"_"</span><span class="p">)</span>
<span class="n">filename</span> <span class="o">=</span> <span class="n">filename</span><span class="p">.</span><span class="n">replace</span><span class="p">(</span><span class="s">","</span><span class="p">,</span> <span class="s">"-"</span><span class="p">)</span>
<span class="k">return</span> <span class="n">filename</span>
<span class="n">addons</span> <span class="o">=</span> <span class="p">[</span><span class="n">EditableCache</span><span class="p">()]</span>
</code></pre></div></div>
<p>About this example :</p>
<ul>
<li>the <code class="language-plaintext highlighter-rouge">url_filter</code> param allows you to only save some of the requests that pass through <strong>mitmproxy</strong></li>
<li>the <code class="language-plaintext highlighter-rouge">path</code> param is the local folder where you want to save responses</li>
</ul>
<hr />
<p>I hope this post will inspire some of you to create their own <strong>mitmproxy</strong> scripts. You can find more examples on my <a href="https://github.com/lucaslegname/mitmproxy-helpers" target="_blank">mitmproxy-helpers</a> Github repository. Don’t hesitate to fork this repo and to suggest improvements!</p>
<h2 id="other-posts-about-mitmproxy">Other posts about mitmproxy</h2>
<p>If you want to learn more about <strong>mitmproxy</strong>, don’t hesitate to check my other posts about this topic :</p>
<ul>
<li><a href="/mitmproxy/2020/04/10/mitmproxy.html">How to inspect network traffic using mitmproxy</a></li>
<li><a href="/mitmproxy/2020/04/22/mitmproxy-tips.html">Useful tips for mitmproxy</a></li>
</ul>Lucas LegnameNow that we covered the first steps to get started with mitmproxy, I would like to develop a bit more one of the best features the tool has to offer : the ability to run Python scripts.Useful tips for mitmproxy2020-04-22T18:20:20+00:002020-04-22T18:20:20+00:00/mitmproxy/2020/04/22/mitmproxy-tips<p>In <a href="/mitmproxy/2020/04/10/mitmproxy.html">my previous post</a>, I focused on steps to install <strong>mitmproxy</strong> and set up your target device. Now that you are all set, here are a few tips to help you getting quickly confortable with this tool.</p>
<h2 id="the-table-and-details-views">The table and details views</h2>
<p>mitmproxy starts on the “table view”, that’s where the requests appear. To navigate among requests, you can use the <strong>up</strong> and <strong>down</strong> keys. Select a line and hit <strong>enter</strong> to see details from any request.</p>
<p>When using an app, this view can quickly become crowded. In this case, you can use the following commands to reduce the number of displayed requests :</p>
<ul>
<li>use <strong>d</strong> to remove a single request</li>
<li>use <strong>z</strong> to clear the list</li>
</ul>
<p><img src="/assets/images/mitmproxy/table-view.jpg" alt="Table view" /></p>
<p>The “request details view” is divided into three tabs (<strong>Request</strong>, <strong>Response</strong> and <strong>Detail</strong>), you can use the <strong>tab</strong> key to switch between tabs.</p>
<p>This view allows you to see the header and body from any request or response. You can also see details about the web-server that responded.</p>
<p><img src="/assets/images/mitmproxy/details-view.jpg" alt="Details view" /></p>
<p>You can navigate between requests directly from the “request details view” : hit <strong>space</strong> to go to the next request and <strong>p</strong> to go to the previous one.</p>
<h2 id="mitmproxy-essential-keys">mitmproxy essential keys</h2>
<p>With the several following keys, you should be able to navigate among views and explore possibilities offered by each of them :</p>
<ul>
<li>wherever you are, use <strong>q</strong> to go the previous screen</li>
<li>the <strong>escape</strong> key allows you to cancel any ongoing modification</li>
<li>to open the <a href="https://docs.mitmproxy.org/stable/concepts-options/" target="_blank">options</a> view, use <strong>O</strong></li>
<li>you can use <strong>?</strong> on any view to see global and view-specific keybindings</li>
</ul>
<p><img src="/assets/images/mitmproxy/keybindings.jpg" alt="Keybindings" /></p>
<h2 id="filter-requests">Filter requests</h2>
<p>You will quickly notice that your phone or tablet constantly sends and receives a lot of data even when you are not actively using it. That is why filtering is necessary when you want to follow requests done for a specific purpose.</p>
<p><strong>mitmproxy</strong> allows you to filter requests <a href="https://docs.mitmproxy.org/stable/concepts-filters/" target="_blank">using regexes</a>. To change the filter, simply hit <strong>f</strong> when you are on the table view.</p>
<p>To display <strong>all requests from google.com</strong> you must use :</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">set </span><span class="nv">view_filter</span><span class="o">=</span>google<span class="se">\.</span>com
</code></pre></div></div>
<p>To display <strong>404 errors from google.com</strong> you must use :</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">set </span><span class="nv">view_filter</span><span class="o">=</span>google<span class="se">\.</span>com ~c 404
</code></pre></div></div>
<h2 id="intercept-requests">Intercept requests</h2>
<p>The same regexes can also be used as “intercept filters” : this feature allows you to put on hold some requests.</p>
<p>Imagine your app crashes when arriving on a specific screen. You notice that 3 different endpoints are called to load this screen’s content and suspect one of the responses to make your app crash.</p>
<p>With the intercept filter you can block the 3 requests and then authorize each one of them individualy : this way you can see which one makes you app crash.</p>
<ul>
<li>use the <strong>i</strong> key to set an “intercept filter”</li>
<li>when a request is intercepted, select it and press <strong>a</strong> to let it go</li>
<li>when the response arrives, it will also be on hold : press <strong>a</strong> to let it pass</li>
</ul>
<h2 id="ignore-domains">Ignore domains</h2>
<p>You might discover at some point that few apps use <a href="https://owasp.org/www-community/controls/Certificate_and_Public_Key_Pinning" target="_blank">certificate pinning</a> to prevent reverse-engineering.</p>
<p>A famous example is the <strong>AppStore</strong> : if you try to open the app while using <strong>mitmproxy</strong>, all requests will be rejected and you won’t see any traffic.</p>
<p>Now imagine that you need to test deferred deep linking for an iOS app : you open a deeplink, reach the store, download the app and open it.</p>
<p>Without the <code class="language-plaintext highlighter-rouge">--ignore-hosts</code> parameter, requests from the <strong>Appstore</strong> app would be blocked. To ignore requests from the store and see others you must use :</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mitmproxy <span class="nt">--ignore-hosts</span> <span class="s1">'(apple\.com|mzstatic\.com):443'</span>
</code></pre></div></div>
<p>Another convenient use case if when you want to totally bypass <strong>mitmproxy</strong> without modifying the proxy settings on the device. In this case, use <code class="language-plaintext highlighter-rouge">':'</code> to ignore all domains :</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mitmproxy <span class="nt">--ignore-hosts</span> <span class="s1">':'</span>
</code></pre></div></div>
<h2 id="export-requests">Export requests</h2>
<p>To reproduce a bug, it is generally useful to export requests to share them with another team for example. <strong>mitmproxy</strong> allows you to export requests using the <strong>e</strong> key.</p>
<p>After selecting the export format (<strong>cURL</strong>, <strong>httpie</strong> or <strong>raw</strong>), you need to enter the export path. If you directly installed the software on you computer, just use any local path.</p>
<p>If you use <strong>mitmproxy</strong> through <strong>Docker</strong>, then you must use a shared folder. For example, the one you use for certificates :</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>: export.file curl @focus /home/mitmproxy/.mitmproxy/request.curl
</code></pre></div></div>
<h2 id="to-go-further">To go further</h2>
<p>With the few tips from this post, you should already be able to do a lot of things. If you want to go further, you can have a look at <a href="https://docs.mitmproxy.org/stable/" target="_blank">mitmproxy’s documentation</a> and at the following posts :</p>
<ul>
<li><a href="/mitmproxy/2020/04/10/mitmproxy.html">How to inspect network traffic using mitmproxy</a></li>
<li><a href="/mitmproxy/2020/11/04/mitmproxy-scripts.html">Creating scripts for mitmproxy</a></li>
</ul>Lucas LegnameIn my previous post, I focused on steps to install mitmproxy and set up your target device. Now that you are all set, here are a few tips to help you getting quickly confortable with this tool.Installing a self-signed certificate on an Android device2020-04-10T18:20:20+00:002020-04-10T18:20:20+00:00/mitmproxy/2020/04/10/certificate-android<p><em>For the purpose of this tutorial, I used a Samsung Galaxy S9 running <strong>Android 9</strong>.</em></p>
<h2 id="lets-mitmit">Let’s mitm.it</h2>
<p>On the phone or tablet, open a web browser and go to the following url :</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>http://mitm.it
</code></pre></div></div>
<p>If the network settings are properly set on your device, you should land on a page looking like this one :</p>
<p><img src="/assets/images/mitmproxy/certificate-install-android-01.jpg" alt="Certificate" /></p>
<p>Hit the Android logo to download the <strong>mitmproxy certificate</strong>. An alert should appear, saying that you are about to download the <code class="language-plaintext highlighter-rouge">mitmproxy-ca-cert.pem</code> file : hit <strong>Download</strong>.</p>
<p>Another alert will ask you to set a password on your device in order to use self-signed certificates, if the device already has a password you will be asked to key it in.</p>
<p>Now you need to choose a certificate name, insert any name then hit <strong>OK</strong>.</p>
<p>On <strong>Android</strong>, certificates installed by users are <strong>enabled by default</strong> so your device should be ready at this point.</p>
<p>If you need to manage certificates you installed, you can find them in <strong>Settings</strong> > <strong>Biometrics and security</strong> > <strong>Other security settings</strong> > <strong>User certificates</strong>.</p>
<hr />
<p>That’s it! The mitmproxy certificate is now installed on your device : let’s go back to the main tutorial and try catching a few requests.</p>
<p><a href="/mitmproxy/2020/04/10/mitmproxy.html#installing-the-self-signed-certificate"><em>< Back to the <strong>mitmproxy</strong> tutorial</em></a></p>Lucas LegnameFor the purpose of this tutorial, I used a Samsung Galaxy S9 running Android 9.Installing a self-signed certificate on an iOS device2020-04-10T18:20:20+00:002020-04-10T18:20:20+00:00/mitmproxy/2020/04/10/certificate-ios<p><em>For the purpose of this tutorial, I used an iPhone running <strong>iOS 13</strong>.</em></p>
<h2 id="lets-mitmit">Let’s mitm.it</h2>
<p>On the phone or tablet, open a web browser and go to the following url :</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>http://mitm.it
</code></pre></div></div>
<p>If the network settings are properly set on your device, you should land on a page looking like this one :</p>
<p><img src="/assets/images/mitmproxy/certificate-install-ios-01.png" alt="Certificate" /></p>
<p>Hit the Apple logo in order to download the mitmproxy certificate. An alert should appear, saying that you are about to download a configuration profile : hit <strong>Allow</strong>.</p>
<p>Another alert will confirm that the profile has been downloaded and that you need to review it before installing it.</p>
<h2 id="installing-the-profile">Installing the profile</h2>
<p>Go to <strong>Settings</strong>, on the main screen you should see a new entry named <strong>Profile Downloaded</strong>.</p>
<p><img src="/assets/images/mitmproxy/certificate-install-ios-02.png" alt="Certificate" /></p>
<p>Open it and a modal allowing you to install the configuration profile (containing the mitmproxy certificate) should appear : hit <strong>Install</strong>. iOS will ask for you password and then you will have to hit <strong>Install</strong> once again.</p>
<h2 id="enabling-the-certificate">Enabling the certificate</h2>
<p>Last but not least, you need to enable the mitmproxy certificate. First, go to <strong>Settings</strong> > <strong>General</strong> > <strong>About</strong>.</p>
<p>Now, at the bottom of the list, hit <strong>Certificate Trust Settings</strong> and enable the mitmproxy certificate.</p>
<p><img src="/assets/images/mitmproxy/certificate-install-ios-03.png" alt="Certificate" /></p>
<p>That’s it! The mitmproxy certificate is now installed on your device : let’s go back to the main tutorial and try catching a few requests.</p>
<p><a href="/mitmproxy/2020/04/10/mitmproxy.html#installing-the-self-signed-certificate"><em>< Back to the <strong>mitmproxy</strong> tutorial</em></a></p>Lucas LegnameFor the purpose of this tutorial, I used an iPhone running iOS 13.Installing a self-signed certificate on a tvOS device2020-04-10T18:20:20+00:002020-04-10T18:20:20+00:00/mitmproxy/2020/04/10/certificate-tvos<p><em>For the purpose of this tutorial, I used an Apple TV 4K running <strong>tvOS 13</strong>.</em></p>
<h2 id="locating-the-mitmproxy-certificate">Locating the mitmproxy certificate</h2>
<p>Depending on how you installed mitmproxy on your computer, the certificate folder should be either <code class="language-plaintext highlighter-rouge">~/.mitmproxy</code> or the local folder you passed as parameter in the <strong>Docker</strong> command.</p>
<p>Once you found the folder, locate the <code class="language-plaintext highlighter-rouge">mitmproxy-ca-cert.cer</code> file among the multiple certificate files. If it is not there, then try starting <strong>mitmproxy</strong> and check once again.</p>
<h2 id="uploading-the-certificate">Uploading the certificate</h2>
<p>We now need to send the certificate to your Apple TV. To do so, we first need to host it somewhere.</p>
<p>You can use <a href="http://dropbox.com" target="_blank">Dropbox</a> for example : send the file in the cloud and get the <strong>download link</strong>. You should get something looking like this :</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>https://www.dropbox.com/s/ppldnxk2531p4ze/mitmproxy-ca-cert.cer?dl=0
</code></pre></div></div>
<p>Just replace the <code class="language-plaintext highlighter-rouge">?dl=0</code> with <code class="language-plaintext highlighter-rouge">?dl=1</code> and you get a link totally usable by your device.</p>
<h2 id="installing-the-certificate">Installing the certificate</h2>
<p>Go to <strong>Settings</strong> > <strong>General</strong> > <strong>Privacy</strong>, select <strong>Share Apple TV Analytics</strong> and hit the <strong>PLAY</strong> button on the remote.</p>
<p><img src="/assets/images/mitmproxy/network-settings-tvos-01.jpg" alt="Download profile" /></p>
<p>You should be in the <strong>Profiles</strong> hidden menu. From there, click on <strong>Add Profile</strong> and enter the url from your hosted certificate.</p>
<p><img src="/assets/images/mitmproxy/network-settings-tvos-02.jpg" alt="Download profile" /></p>
<p>Once you are done, hit <strong>Done</strong> and the certificate should be added to your Apple TV.</p>
<h2 id="enabling-the-certificate">Enabling the certificate</h2>
<p>Last but not least, you need to enable the mitmproxy certificate. First, go to <strong>Settings</strong> > <strong>General</strong> > <strong>About</strong>, select the <strong>Certificate Trust Settings</strong> sub-menu and hit <strong>OK</strong>.</p>
<p><img src="/assets/images/mitmproxy/certificate-install-tvos-01.jpg" alt="Enable certificate" /></p>
<p>Here you should see your freshly added mitmproxy certificate. To trust it, just select the certificate, click <strong>OK</strong> then <strong>Continue</strong>. Once tvOS mentions <strong>Trusted</strong> next to the certificate, you are good.</p>
<p><img src="/assets/images/mitmproxy/certificate-install-tvos-02.jpg" alt="Enable certificate" /></p>
<p>That’s it! The mitmproxy certificate is now installed on your device : let’s go back to the main tutorial and try catching a few requests.</p>
<p><a href="/mitmproxy/2020/04/10/mitmproxy.html#installing-the-self-signed-certificate"><em>< Back to the <strong>mitmproxy</strong> tutorial</em></a></p>Lucas LegnameFor the purpose of this tutorial, I used an Apple TV 4K running tvOS 13.How to inspect network traffic using mitmproxy2020-04-10T18:20:20+00:002020-04-10T18:20:20+00:00/mitmproxy/2020/04/10/mitmproxy<p>Most of modern apps rely on APIs. « <em>- How can I access to your catalog ? - Here’s an API for that.</em> » APIs are everywhere and that’s not about to change.</p>
<p>Breaking down each part of your code into separate services is <em>(often)</em> good but when it comes to testing, this separation might make things a bit harder.</p>
<p>Front-end apps developers do not write anymore the code that takes care of the core business logic. Generally, complex data-processing tasks are done on API-side and front-end apps just have to display what is provided by the API.</p>
<p>This is why it is so important to be able to retrace the data path from the app to the API. When you test and validate an app, being able to answer the following questions is crucial :</p>
<ul>
<li>what API-endpoint(s) is the app using for this screen ?</li>
<li>what is contained in the response provided by the API ?</li>
<li>what happens if the API response structure changes ?</li>
</ul>
<p>By the end of this tutorial you should be able to use <strong><a href="https://mitmproxy.org" target="_blank">mitmproxy</a></strong> to answer those questions. But you will see, there are many more things to do with this amazing tool.</p>
<p><img src="/assets/images/mitmproxy/how-mitmproxy-works.jpg" alt="How mitmproxy works" /></p>
<h2 id="before-we-start">Before we start…</h2>
<p>You might be wondering why you should use <strong>mitmproxy</strong> instead of another tool. And that’s a good question (<em>don’t follow the hype!</em>).</p>
<p>My first argument is the price : <strong>mitmproxy</strong> is free (released under <strong>MIT Licence</strong>) when a licence for <strong>Charles Proxy</strong> costs around 50$.</p>
<p>Also <strong>mitmproxy</strong> is a very powerful tool : not only it provides options to rewrite traffic but it also allows you to run <a href="https://docs.mitmproxy.org/stable/addons-scripting/" target="_blank">Python scripts</a> to alter requests and responses. This is really helpful when trying to test edge-cases.</p>
<p>To finish, <strong>mitmproxy</strong> can run anywhere with a web interface or in command line : no matter the context, there are good chances you will be able to use it.</p>
<h2 id="installation">Installation</h2>
<p>There are multiple ways to install <strong>mitmproxy</strong>. Using <strong>brew</strong> is the « <em>standard</em> » way but I personally prefer using it through <strong>Docker</strong> : it prevents me from dealing with Python versions and also helps to keep my Mac clean. Moreover, it’s really easy to do and no particular <strong>Docker</strong> knowledge is required.</p>
<p>For the installation, feel free to choose any method you feel the most comfortable with.</p>
<h3 id="using-brew">Using brew</h3>
<p>If you already have <strong>brew</strong> installed on your computer, and if everything goes well, running this simple command should be enough to do the job :</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>brew install mitmproxy
</code></pre></div></div>
<p>Once <strong>mitmproxy</strong> is installed you can directly jump to the <a href="#lets-run-mitmproxy">Let’s run mitmproxy</a> section.</p>
<h3 id="using-docker">Using Docker</h3>
<p>Running <strong>mitmproxy</strong> using <a href="https://hub.docker.com/r/mitmproxy/mitmproxy/" target="_blank">the official Docker image<a></a> is as simple as executing the following command :</a></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker run <span class="nt">--rm</span> <span class="nt">-it</span> <span class="nt">-p</span> 8080:8080 mitmproxy/mitmproxy:4.0.4
</code></pre></div></div>
<p>To avoid having to install <strong>mitmproxy</strong>’s certificates on your test devices every time, it is strongly advised to share the container’s certificate folder with a local one (using the <code class="language-plaintext highlighter-rouge">-v</code> param).</p>
<p>This way, certificates will be persisted even after your <strong>Docker</strong> container is killed.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker run <span class="nt">--rm</span> <span class="nt">-it</span> <span class="nt">-v</span> /your/local/folder:/home/mitmproxy/.mitmproxy <span class="nt">-p</span> 8080:8080 mitmproxy/mitmproxy:4.0.4
</code></pre></div></div>
<p>To simplify the use of this command, add an <strong>alias</strong> in your <code class="language-plaintext highlighter-rouge">~/.bash_profile</code> :</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">alias </span><span class="nv">mitmproxy</span><span class="o">=</span><span class="s1">'docker run --rm -it -v /your/local/folder:/home/mitmproxy/.mitmproxy -p 8080:8080 mitmproxy/mitmproxy:4.0.4'</span>
</code></pre></div></div>
<p>And don’t forget to execute <code class="language-plaintext highlighter-rouge">source ~/.bash_profile</code> after you edited the file.</p>
<h2 id="lets-run-mitmproxy">Let’s run mitmproxy</h2>
<p>No matter how you installed it, you should now be able to start <strong>mitmproxy</strong> with the command :</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mitmproxy
</code></pre></div></div>
<p>If the program starts then congrats! You can keep going!</p>
<p><img src="/assets/images/mitmproxy/mitmproxy-empty.jpg" alt="mitmproxy is alive" /></p>
<h2 id="network-settings-on-the-target-device">Network settings on the target device</h2>
<p><strong>mitmproxy</strong> can run as a <a href="https://docs.mitmproxy.org/stable/concepts-modes/#reverse-proxy" target="blank">reverse proxy</a>, as a <a href="https://docs.mitmproxy.org/stable/concepts-modes/#transparent-proxy" target="blank">transparent proxy</a> or as a <a href="https://docs.mitmproxy.org/stable/concepts-modes/#regular-proxy" target="blank">regular proxy</a>. To keep things simple, we will use it as a regular proxy.</p>
<p>Using the program this way is certainly easier but it means that you can change the network settings of the the target device in order to set a proxy.</p>
<p>This is really simple when you need to inspect requests from an <strong>iOS</strong> or <strong>Android</strong> device but gets more complicated with devices that do not provide advanced settings for network.</p>
<p>Depending on your target device, just follow the corresponding sub-tutorial :</p>
<ul>
<li><a href="/mitmproxy/2020/04/10/setting-proxy-ios.html">setting a proxy on an iOS device</a></li>
<li><a href="/mitmproxy/2020/04/10/setting-proxy-android.html">setting a proxy on an Android device</a></li>
<li><a href="/mitmproxy/2020/04/10/setting-proxy-tvos.html">setting a proxy on a tvOS device</a></li>
</ul>
<h2 id="installing-the-self-signed-certificate">Installing the self-signed certificate</h2>
<p>At this moment, if you start <strong>mitmproxy</strong> your device will accept HTTP traffic but reject any HTTPS request not coming directly from the server. The reason is simple : your device won’t accept any <strong>self-signed certificate</strong> unless you ask it to do so.</p>
<p>To intercept HTTPS requests, we need to set the certificate used by <strong>mitmproxy</strong> on the target device. This will allow your device to accept secured requests coming from your computer instead of the real server.</p>
<p>The complexity of installing a self-signed certificate really depends on the nature of the device you need to inspect. Basically, it goes from « <em>very simple</em> » to « <em>not possible</em> ».</p>
<p>Depending on your target device, just follow the corresponding sub-tutorial :</p>
<ul>
<li><a href="/mitmproxy/2020/04/10/certificate-ios.html">installing a self-signed certificate on an iOS device</a></li>
<li><a href="/mitmproxy/2020/04/10/certificate-android.html">installing a self-signed certificate on an Android device</a></li>
<li><a href="/mitmproxy/2020/04/10/certificate-tvos.html">installing a self-signed certificate on a tvOS device</a></li>
</ul>
<p><img src="/assets/images/mitmproxy/certificate.jpg" alt="Certificate" /></p>
<h2 id="lets-intercept-calls">Let’s intercept calls</h2>
<p>Once the certificate is installed, you are ready to intercept calls. You can start <strong>mitmproxy</strong> once again using :</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mitmproxy
</code></pre></div></div>
<p>On the target device, open an app and check if new lines appear in the terminal.</p>
<p><img src="/assets/images/mitmproxy/mitmproxy.jpg" alt="mitmproxy" /></p>
<p>If you see requests, then congrats! Everything is set up and you can start inspecting the web traffic going through your phone or tablet.</p>
<hr />
<h2 id="troubleshooting">Troubleshooting</h2>
<p>If nothing appears and the target device seems to have no internet connection, here are a few things you should check :</p>
<ul>
<li>are both the computer running <strong>mitmproxy</strong> and the target device on the same network ?</li>
<li>have you tried restarting <strong>mitmproxy</strong> ?</li>
<li>are the <strong>proxy settings</strong> correctly set on the target device ?</li>
<li>have you enabled the <strong>self-signed certificate</strong> on the target device ?</li>
</ul>
<h2 id="to-go-further">To go further</h2>
<p>Now you should know everything you need to install <strong>mitmproxy</strong> and configure your target device. If you want to go further, have a look at the following posts :</p>
<ul>
<li><a href="/mitmproxy/2020/04/22/mitmproxy-tips.html">Useful tips for mitmproxy</a></li>
<li><a href="/mitmproxy/2020/11/04/mitmproxy-scripts.html">Creating scripts for mitmproxy</a></li>
</ul>Lucas LegnameA quick introduction to install mitmproxy and set up the device you would like to inspect. By the end of this tutorial you should be able to see HTTPS requests from any iOS, Android or tvOS device.Setting a proxy on an Android device2020-04-10T18:20:20+00:002020-04-10T18:20:20+00:00/mitmproxy/2020/04/10/setting-proxy-android<p><em>For the purpose of this tutorial, I used a Samsung Galaxy S9 running <strong>Android 9</strong>.</em></p>
<p>Before you start, make sure that both the target device and the proxy (the computer running <strong>mitmproxy</strong>) are on the same network.</p>
<p>On the target device, open <strong>Settings</strong> > <strong>Connections</strong> > <strong>Wi-Fi</strong>, show informations from your current network (hit the network you are already connected to) and select <strong>Advanced</strong>.</p>
<p>Change the proxy setting to <strong>Manual</strong> and set the following values :</p>
<ul>
<li>Proxy host name : <em>the local IP address of your computer running <strong>mitmproxy</strong></em></li>
<li>Proxy port : <em>8080</em></li>
<li><em>Leave other fields blank or with default value</em></li>
</ul>
<p><img src="/assets/images/mitmproxy/network-settings-android-01.jpg" alt="Android settings" /></p>
<p>To get the local IP address of the computer serving as proxy, generally the easiest way is to open its network settings (<strong>Settings</strong> > <strong>Network</strong> on <strong>MacOS</strong>) and look for its local IP address, <em>192.168.1.42</em> in this example :</p>
<p><img src="/assets/images/mitmproxy/network-ip-macos.jpg" alt="MacOS network settings" /></p>
<p><a href="/mitmproxy/2020/04/10/mitmproxy.html#network-settings-on-the-target-device"><em>< Back to the <strong>mitmproxy</strong> tutorial</em></a></p>Lucas LegnameFor the purpose of this tutorial, I used a Samsung Galaxy S9 running Android 9.Setting a proxy on an iOS device2020-04-10T18:20:20+00:002020-04-10T18:20:20+00:00/mitmproxy/2020/04/10/setting-proxy-ios<p><em>For the purpose of this tutorial, I used an iPhone running <strong>iOS 13</strong>.</em></p>
<p>Before you start, make sure that both the target device and the proxy (the computer running <strong>mitmproxy</strong>) are on the same network.</p>
<p>On the target device, open <strong>Settings</strong> > <strong>Wi-Fi</strong>, show informations from your current network (hit the small <strong><em>i</em></strong> icon) and select <strong>Configure Proxy</strong>.</p>
<p>Use <strong>Manual</strong> and set the following values :</p>
<ul>
<li>Server : <em>the local IP address of your computer running <strong>mitmproxy</strong></em></li>
<li>Port : <em>8080</em></li>
<li>Authentication : <em>keep disabled</em></li>
</ul>
<p><img src="/assets/images/mitmproxy/network-settings-ios-01.png" alt="SetApp" /></p>
<p>To get the local IP address of the computer serving as proxy, generally the easiest way is to open its network settings (<strong>Settings</strong> > <strong>Network</strong> on <strong>MacOS</strong>) and look for its local IP address, <em>192.168.1.42</em> in this example :</p>
<p><img src="/assets/images/mitmproxy/network-ip-macos.jpg" alt="MacOS network settings" /></p>
<p><a href="/mitmproxy/2020/04/10/mitmproxy.html#network-settings-on-the-target-device"><em>< Back to the <strong>mitmproxy</strong> tutorial</em></a></p>Lucas LegnameFor the purpose of this tutorial, I used an iPhone running iOS 13.Setting a proxy on a tvOS device2020-04-10T18:20:20+00:002020-04-10T18:20:20+00:00/mitmproxy/2020/04/10/setting-proxy-tvos<p><em>For the purpose of this tutorial, I used an Apple TV 4K running <strong>tvOS 13</strong>. Please note that since <strong>Apple Configurator</strong> is necessary to set a proxy on Apple TV, a Mac is needed to realize the following steps.</em></p>
<p>The first thing to know when you want to set a proxy on an Apple TV is that there is no sub-menu in the <strong>tvOS</strong> settings allowing you to do so.</p>
<p>Fortunately, it is possible to proceed using a configuration profile created with <strong>Apple Configurator 2</strong>. You can download this tool from the <a href="https://apps.apple.com/us/app/apple-configurator-2/id1037126344?mt=12" target="_blank">Appstore</a>.</p>
<h2 id="creating-the-configuration-profile">Creating the configuration profile</h2>
<p>Start <strong>Apple Configurator</strong> and create a new profile (<strong>File</strong> > <strong>New Profile</strong>). Select the <strong>Wi-Fi</strong> section on the left then hit <strong>Configure</strong>.</p>
<p>Enter the name of your network (SSID), the security type (WEP/WPA2/…) and the password of your wireless network.</p>
<p>Then set the <strong>Proxy Setup</strong> to <strong>Manual</strong> and use the following values :</p>
<ul>
<li>Host name : <em>the local IP address of your computer running <strong>mitmproxy</strong></em></li>
<li>Port number : <em>8080</em></li>
<li><em>Leave other fields blank or with default value</em></li>
</ul>
<p><img src="/assets/images/mitmproxy/apple-configurator-wifi.jpg" alt="Apple Configurator 2" /></p>
<p>To get the local IP address of the computer serving as proxy, generally the easiest way is to open its network settings (<strong>Settings</strong> > <strong>Network</strong> on <strong>MacOS</strong>) and look for its local IP address, <em>192.168.1.42</em> in this example :</p>
<p><img src="/assets/images/mitmproxy/network-ip-macos.jpg" alt="MacOS network settings" /></p>
<p>Save this new configuration profile somewhere : the created file should have the <code class="language-plaintext highlighter-rouge">.mobileconfig</code> extension.</p>
<h2 id="uploading-the-file">Uploading the file</h2>
<p>We now need to send the configuration profile to your Apple TV. To do so, we first need to host it somewhere.</p>
<p>You can use <a href="http://dropbox.com" target="_blank">Dropbox</a> for example : send the file in the cloud and get the <strong>download link</strong>. You should get something looking like this :</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>https://www.dropbox.com/s/ppldnxk2531p4ze/atv.mobileconfig?dl=0
</code></pre></div></div>
<p>Just replace the <code class="language-plaintext highlighter-rouge">?dl=0</code> with <code class="language-plaintext highlighter-rouge">?dl=1</code> and you get a link totally usable by your device.</p>
<h2 id="installing-the-profile">Installing the profile</h2>
<p>Go to <strong>Settings</strong> > <strong>General</strong> > <strong>Privacy</strong>, select <strong>Share Apple TV Analytics</strong> and hit the <strong>PLAY</strong> button on the remote.</p>
<p><img src="/assets/images/mitmproxy/network-settings-tvos-01.jpg" alt="Download profile" /></p>
<p>You should get in the <strong>Profiles</strong> hidden menu. From there, click on <strong>Add Profile</strong> and enter the url from your hosted configuration profile.</p>
<p><img src="/assets/images/mitmproxy/network-settings-tvos-02.jpg" alt="Download profile" /></p>
<p>Once you are done, hit <strong>Done</strong> and the profile should be added to your Apple TV. You might need to reboot your device in order to get the profile applied.</p>
<p><a href="/mitmproxy/2020/04/10/mitmproxy.html#network-settings-on-the-target-device"><em>< Back to the <strong>mitmproxy</strong> tutorial</em></a></p>Lucas LegnameFor the purpose of this tutorial, I used an Apple TV 4K running tvOS 13. Please note that since Apple Configurator is necessary to set a proxy on Apple TV, a Mac is needed to realize the following steps.