Created: 2012-03-26 18:57
Updated: 2017-07-09 06:47

Twitter URL Regexen

This is an attempt to extract the URL-matching regular expression(s) from the twitter-text-rb project and port them to Python.

What? Why?

This is important because any service that will be creating tweets on behalf of its users needs to be able to provide those users an accurate character count. Should be easy, right? Nope! Now that Twitter is wrapping URLs in short links, the service in question will need to know exactly what parts of a user's tweet will be replaced by shortened URLs.

This is mind-bendingly stupid.

To alleviate some of the pain, Twitter has kindly provided reference implementations for this behavior for Ruby, JavaScript, and Java, along with a suite of conformance tests for third party implementers.

Unfortunately, they do not provide and maintain a reference implementation for Python, and those that exist are incomplete (this particular little lib very much included).

Okay, so what is this then?

This project, twitter-url-regexen, is just an attempt to extract the URL-related regular expressions from the twitter-text-rb source code (specifically, lib/regex.rb) and port them to Python. Nothing more, nothing less.

How good is this port?

Well, it passes 62 of the 70 (at time of writing) URL-extraction conformance tests. Hopefully, that's good enough for government work (as they say).

What about those 8 failing tests?

See, here's where things get even dumber. Twitter is not just extracting these URLs based on a regular expression. There is also a fair amount of extra work done to, e.g., specially handle URLs, protocol-less ccTLD URLs, non-ASCII URLs, etc.

This library does not attempt to duplicate that logic, so some of the failures stem from that basic incompatibility.

Fuck if I know about the other ones.

Running the tests

First, get the conformance tests:

git submodule update --init

Make sure you have all of the requirements (py.test and pyyaml):

pip install -r test_requirements.txt

Run the tests:


If you'd like to test some alternate Python implentations (twitter-text-python and twitter-text-py) of the same idea, you'll have to run those tests a little more manually:

py.test tests/alt*

See Also

Twitter's own docs about this inane misfeature:

Cookies help us deliver our services. By using our services, you agree to our use of cookies Learn more