Arrow and function keys are mapped to escape sequences, so if you use vi over a slow or unreliable connection your arrow keys could randomly get interpreted as escape followed by some commands.
timeoutlen is also used by leader key combos, and things like gg, fx, etc. If you drop your timeoutlen, make sure you can still combo these fast enough for them to register.
