Hey, I'm Konur

KonurPapa.com


I tend to be a "paralysis by analysis" kind of guy. I've always been that way, and it's been a challenge I've actively tried to work on over the years.

One practical tool that's helped me tons is giving myself a set amount of time to process a decision (maybe end of day/week), and then by the end of that time, just pulling the trigger. Usually it doesn't take as long as it feels like it does to fully assess all the facts. There's no reason to overthink it.

After that time, I'm not allowed to let my mind wander to all the "what ifs" and "maybes". I've already made the choice, so now I move forward from there. There's a time and a place for reflection, but it does me no good to dwell on something I can no longer change. I almost never discover any new, revelatory information after that time that would have completely changed my decision, anyway.

There's a good phrase:

> "Don't worry about making the wrong choice. Just make a choice, and then make that choice the right one."

This is 100% the mindset to have. Even the best decisions will sometimes turn out horribly, and it does no good to feel guilty on top of it if I truly did my best.

Here's what I always come back to: I'll be less upset with myself for making a poor choice than for being too scared to make a choice at all.

Moving past the fear of making the wrong choice

Making an Important Decision


It's easy to be selfish in conversation. "Will this person shut up already so I can say what I want to say?"

This is not a way to have meaningful conversation. People are not as stupid as we think; they can see through our disinterest. Here are a few thoughts on what I try to cultivate in myself when talking with people:

I often remind myself that (most likely) I know very little about this person's viewpoint on the particular topic of conversation. Not only is it good to foster curiosity about how they see things so I can get to know them better, but I will look pretty foolish if I jump to conclusions and don't hear them out (this goes for most things -- be it confrontation, preferences, or otherwise).

My general rule of thumb is I want the other person to leave the conversation thinking, "wow, I like talking to Konur!" Even if we disagree on a subject, there's no reason the conversation can't be an enjoyable one. If the other person has a good impression about me when we leave the conversation, they will want to talk to me more in the future -- and more importantly trust will be built.

Trust is the bedrock upon which (possibly all?) meaningful interaction stems from. As such, this means that learning how to communicate in this way is one of the most vital skills one can learn.

An Approach to Conversation


Ever wondered how spellcheck works? Perhaps if you've ever been angry by its incorrect rewording of whatever it was you wrote.

But consider for a moment how often it gets things _right_. How does it so often seem to know what word you meant to type, based on your typo?

My latest nerdy forray has been into the world of Levenshtein automata. Simply put, this is an algorithm for determining how similar a given string of text is to another string (probably from a list). Like the spellcheck example: how similar is the typo `stea` to an actual word in the dictionary?

> This is an algorithm for determining how similar a given string of text is to another.

In this example, there are some things to consider. Firstly, this "similarity" can be composed of 3 distinct logical operations: deletions, insertions, and replacements. In this case, `stea` could be matched within 1 operation to the words `tea` (a deletion), `stead` (an insertion), and `teas` (a replacement).

This is why spellcheck often offers more than one potential word as a typo solution, and the likelihood of `x` match is based on probability. This is also why some editors don't ask your permission and automatically correct some words; it's certain the word must have been a typo because the probability is so high.


## Beyond Spellcheck

I've been working on a project where I've put this concept to use (hence my deep-dive). I have an input item (or _items_), and I need to find the best match from an internal database of items.

Problem #1 is this input item may or may not be typoed. Problem #2 is this searching database is pretty big (a few thousand entries).

Obviously, the speed at which this processes is of utmost importance. I could manually iterate over every single index in the database until a match is found, which I know would _technically_ work. But if there are numerous input items, this will start to take a really long time:

| Processing Time | x100 Items |
| --------------- | ---------- |
| 0.5 seconds     | 50 seconds |
| 3 seconds       | 5 minutes  |

Once it starts taking even a fraction of a second longer, the processing time is going to multiply out exponentially by the number of items.

I proceeded to do a ton of research, and came across a few good resources. I'd be amiss not to share some of them; specifically, [this](http://norvig.com/spell-correct.html "How to Write a Spelling Corrector") article on the logic behind a spelling corrector, and [this](https://julesjacobs.com/2015/06/17/disqus-levenshtein-simple-and-fast.html "Levenshtein automata can be simple and fast") article explaining how Lenvenshtein automata works (with some Python examples as well).

The solution I eventually came up with is to first try direct key matching by converting all database entries into key/value pairs. Then something like:

```python
input_handler(item):
    return database.get(item) != null
```

This handles a pretty sizeable percentage (apparently people sometimes _can_ spell).

But what about typos? This is where I use Levenshtein automata (AKA "fuzzy" matching). There's a really helpful Python library called `fuzzywuzzy` (I think now it's called `thefuzz`) that has a bunch of built-in functions for handling matching operations. It also returns a probability score, which I use to sort more likely matches from less likely ones.

Here's another helpful article explaining how each of these functions work:

https://www.datacamp.com/tutorial/fuzzy-string-python

So here's my flow of logic:

- Split the database into categories, so we only have to iterate over database entries of the same type (I'll probably get into the details of how this part works in a future post).
- After the direct-key matching attempt, iterate over the entire database list of the same category as the input item.
- Check if the item shares any starting letters in common with the current database item; if not, skip it (no sense wasting time on fuzzy matching).
- Add this potential entry into a list, and rank the list based on similarity (using the built-in Python function `.intersection`).
- Finally, try to perform a fuzzy match on the top potentials.

The resulting system works accurately and runs pretty fast, averaging at about 0.5 seconds (from what I can tell... it's difficult for me to get a totally accurate metric for other reasons).

There are tons of other fun applications for Levenshtein automata; this is just my own example. Hopefully I've explained this in a way that's a little easier to understand and more accessible.

A good tool to have in the programmer arsenal!

Blog

Making an Important Decision

An Approach to Conversation

Understanding Levenshtein Automata