TL;DR: Encoding is hard.

Fixed a dupe detection bug in my bot by using an edit distance limit instead of relying on string equality.
x.com/ianisqatsibot/…
Ian Brown @igb
Twitter's alt text field seems to support certain ws characters that get stripped when posting a tweet containing those chars.
View on Twitter ↗
0 0

Replies

So, being too lazy to figure out how to scrub the source data I added a filter based on an edit distance < 10 to remove likely dupes.
It has an added benefit in this particular bot's case of eliminating similar descriptions of similarly annotated photos.
(which is actually common when a tweet contains multiple photos...often times they are of the same subjects from a different angle or moment)