.tip

sort -u reorders the input. This keeps the first occurrence of each line in the order they arrived.

cat urls.txt | awk '!seen[$0]++'

seen[$0]++ is zero the first time a line appears, so ! makes it true and awk prints the line. The second time it is one, and nothing prints.